Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper suggests a new approach to the construction of ensembles of deep neural networks (DNN). Unlike previous methods which usually deal with multiple DNNs of same structure authors propose to form an ensemble of networks with different architecture. The main claim is that using diverse architectures increases diversity and hence the quality of predictions. To find the best architectures they use methodology inspired by neural architecture search (NAS) in particular random search and regularized evolution. The method for neural ensemble search (NES) is algorithmically simple although computationally hard. On several experiments the authors show NES outperforms standard deep ensembles formed from networks with same (even optimal) structure both in terms of test NLL and in terms of uncertainty estimation under domain shift.

Pros.
Nice idea
Simple algorithm


Cons.
My main point for the criticism is the lack of experiment which I find to be crucially important namely the comparison aganist deep ensemble of DNNs with same architecture to which ForwardSelect procedure has been applied. Train P DNNs with same architecture then perform ForwardSelect routine to take the best K of them and compare your method with such deep ensemble. Currently the authors only compare their method with deep ensembles to which no special selection procedure was applied. This causes bias and it is not clear whether the improvement in NES is due to the usage of different architectures or due to the selection procedure which encourages diversity in resulting ensemble.

P.S. Please correct me if I misunderstood the last point. I have read the corresponding part twice and found no evidence that you're using ForwardSelection when analysing the performance of ensembles of DNNs with same architecture.  

====UPDATE===
My concerns were partly addressed in author's response so I have raised my score to 5.
review_title: Simple and interesting method but the important experiment is missing

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper explores whether one can use Architecture Search to enhance ensemble diversity. They start with the observation that embeddings generated by different architectures (for multiple different initialization per architecture) are well separated from each other. They then try out a couple of architecture search methods to find ensembles with diverse architectures that minimize the loss.

While the novelty is incremental, I like the idea in general. My main objection is that critical baselines are not compared with. Ensemble diversity is a well explored topic with multiple easy to implement regularizations to increase diversity [1, 2, 3] and several more that should be compared with.

[1] “Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensembles” by S Jain, G Liu, DK Gifford (AAAI 2020)
[2] “Ensemble learning via negative correlation” by Y. Liu, X. Yao (Neural Networks 1999)
[3] “Uncertainty in Neural Networks: Approximately Bayesian Ensembling” by Pearce et al (AISTATS 2020) 
review_title: Interesting idea but more experimentation needed

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes creating diverse ensembles of neural networks using an evolutionary method for finding base learners with high performance as well as mutual diversity. The selected base learners are then aggregated for an ensemble using a known method for ensemble selection. The paper is generally well written and addresses a relevant problem of constructing ensembles while training neural networks instead of building models first, independently, and later constructing ensembles. 

Having said that, the paper lacks a significant contribution. The second phase (Ensemble Selection) of the proposed method is essentially the algorithm from Rich Caruana et al. 2004. The first phase of Pool Building suggests either a random generation or alternatively a vaguely described evolutionary method, lacking details or analysis.  It is not clear how exactly is the random initialization of architectures performed. Do you randomly select from a set of seed architectures or randomly create (i.e., random number of layers, random number of units, random activation functions, random initial weights, etc.)?

Growing from a random neural architectures to multiple highly performing (besides being mutually diverse) through single permutations upon model training and evaluation seems like an expensive process. An evolutionary approach in such a manner seems in efficient. Can you report the time taken for some of the reported cases in the evaluation? The ensemble methodology of unweighted averaging is fairly naive. What was the reason to select this one particularly? 

Contribution #1 (page 2) isn't really a contribution. It is common knowledge amongst practitioners. The proposed method can lead to overfitting because the search seems to be based on a fixed set for evaluation. 

Regarding evaluation -- Can you explain how that is addressed? Have you evaluated your method on a broader variety of datasets? Can you confirm that the test data used for search/optimization is different than the one used for measuring reported performance? Did you consider comparing it other methods such as Tao 2019 (mentioned below) ?

This paper can improve its literature survey by citing more directly relevant work in ensemble search using diversification. Here are few examples of more sophisticated ensemble evolution work, not necessarily for a DL base learner, but relevant nonetheless:
-Bhowan, et al. 2013. Evolving diverse ensembles using genetic programming for classification with unbalanced data. Trans. Evol. Comp
-Khurana et al. 2018. Ensembles with Automated Feature Engineering. AutoML at ICML.
-Olson et al. 2019. TPOT: A Tree-Based Pipeline Optimization Tool for Automating Machine Learning. Automated ML. 
-Tao, 2019. Deep Neural Network Ensembles. Machine Learning, Optimization, and Data Science. 
-Yao et al., 2008. Evolving artificial neural network ensembles, in IEEE Computational Intelligence Magazine

Overall, it is a good problem, but this paper falls well short of the threshold.
Update: I thank the authors for their response. Some justifications are provided and for that I will change my score. Overall, the paper still needs work.
review_title: Important problem but the solution lacks novelty 

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors addressed my concerns in the rebuttal. I have raised my score.

Summary:

This paper combines AutoML techniques and deep ensembles to improve the ensemble diversity so that it improves the entire ensemble quality in both in- and out-of-distribution dataset. The authors made a study on two possible AutoML methods which can be combined with ensembles: 1. Random search & 2. Regularized mutation. The empirical results showed that the proposed NES methods outperform commonly selected baselines.

Pros:

How to improve ensemble diversity is one of the core topics on the way to better ensemble performance (in terms of both accuracy and uncertainty metrics). Many previous research works focused on 1. Build efficient ensembles while remaining reasonable diversity; 2. Improve ensemble diversity by exploring the hyper-parameter space. In practice, it is standard to ensemble neural networks with different depths or widths to improve diversity and hence the ensemble performance. However, as far as I know, there is no guidance or automatic mechanism in ensembling neural networks with different architectures. Thus, the problem this paper aims to tackle is significant and it will benefit the research community.

The authors did a self-contained introduction on ensembles & uncertainty and AutoML. The coverage on mutation AutoML is limited but this is still reasonable due to the page constraint. The motivation of why we want to combine AutoML and deep ensembles is clearly stated in section 3.2. Figure 1 demonstrates the effectiveness of various architectures in promoting ensemble diversity.

The empirical evaluation ranges across CIFAR dataset and ImageNet. It also includes calibration performance under dataset shift (uncertainty estimation on out-of-distribution dataset), which is the common benchmark to evaluate an ensemble's performance. The authors made comparisons to several reasonable baselines. The improvement of the proposed NES-RE/RS method over fixed architecture ensembles is consistent and significant. 


Cons:

Figure 1 demonstrates the motivation behind this work. To be more convincing, it can be supplemented with the predictive disagreement on the testset or the averaged KL divergence between the predictive distribution among ensemble members. Moreover,  the figure compares diversity between ensembles with different architectures and ensembles with random seeds. For a more comprehensive study, the figure can include a study on ensembles with different hyper-parameters. A more interesting baseline I will mention below is an ensemble with different depths. 

The baselines considered in this paper only include ensembles with a fixed architecture. It would be more convincing if the authors can include other baselines which include ensembles with different architectures (without neural architecture search). For example, one naive baseline would be ensembling DeepEns(Optimal) with different depths (fully trained independently). This highlights the need for neural architecture search. It also helps to understand how much diversity in architecture (among ensemble members) we need in order to achieve desired diversity in ensemble predictions. Additionally, it is encouraged to compare to hyper-parameter ensembles. This uncovers the question of which axis (hyper-parameter & architectures) is more effective in promoting an ensemble’s performance.

Another missing part in this paper is the cost analysis of NES and how much does it increase compared to deep ensembles. Both NES-RS and NES-RE require training after sampling one neural architecture. This leads to a non-trivial computational overhead compared to traditional deep ensembles. 

In section 4.3, it mentioned that a proportion of validation data encapsulates the belief about test-time shift. The authors didn’t mention whether the same protocol is applied to baselines (DeepEns). This leads to a question that is the improvement on out-of-distribution calibration coming from NES or shifted validation set. I consider this is a minor issue because Figure 21 in the appendix also shows that the clear improvement without shifted validation data.

Table 1 shows that larger ensemble size leads to worse ensembling performance. This is against our general intuition in deep ensembles where more ensemble members lead to better performance. My guess is more ensemble members leads to optimization difficulties in NES. I expect to see more discussion on this observation.

Overall, the authors propose a compelling solution to automatically design the neural network architectures in deep ensembels. However,  the cons slightly outweight the pros in this version.
review_title: A promising solution in an interesting topic. Some modifications are needed to be more convincing. 

Meta-review: This paper proposes a new method to perform uncertainty estimation based on ensembles with diverse network architecture. 

The reviewers raised a few concerns:
- Although it is ok not to compare with (Tao, 2019), an active analytical comparison with baselines for ensemble diversification should not be overlooked e.g. (Yao et al, 2008), (Olson et al, 2019), (Khurana et al, 2018), etc.
- The approach presented in this paper is not novel in the general idea of searching for or diversifying ensembles 
- The reviewers agree that diversity methods can be implemented on top of NES, but it is unclear whether NES+diversity methods would give more over just diversity methods; so either measuring NES+diversity methods or a direct comparison of NES and diversity methods is important.

We encourage the authors address these issues in the next revision.

Decision: Reject