Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This works present a method for hyperparameter optimization applied to ensemble construction. The proposed method builds a surrogate function for diversity of pairs of classifiers that is then combined with a predictive performance objective and maximized. Experiments show that the method outperforms other ensemble building approaches in the literature.
strengths_and_weaknesses: Strengths:
- The contributions are incremental, inscribed in the already existing framework of ensemble learning through hyperparameter optimization
- The paper is well written and structured
- The quality of the experiments is good

Weaknesses:
- The improvement in performance in comparison with the second best method RB-ES is hinting towards a plateau in performance in terms of ensembling

questions: The paper and method description are clear and self-contained. The question is are the contributions significant enough to be published at Neurips. The performance improvement is small between DiVBO and RB-ES but the experiment compares a wide number of approaches and showcases RB-ES itself I believe for the first time. Find some questions/suggestions below. 

- I don't think showing the ranking of validation errors on Figure 3 is very informative, especially given the small size of some of the datasets -- overfitting is likely happening. You should show the ranking of testing error and push back the validation plot to the appendix. Same thing goes for Figure 4 (or maybe reduce number of methods displayed, prune the worse performing methods such as EO, NES and RS-ES and show both validation and testing errors). 

- lines 177-179: are the configurations sampled once per objective function? how many points are sampled?

- line 235: we slightly modifies -> modify

- I think section titles 4.2 and 4.3 should be evaluation OF DivBO/Diversity surrogate and not evaluation on?
limitations: There is a limited discussion of limitations of the method. There are no potential negative societal impacts warranting a discussion.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper focuses on the problem of CASH (Combined Algorithm Selection and Hyperparameter Optimization) to automatically configure machine learning models (or pipelines), and seeks to improve the quality of the final ensemble generated by various AutoML solutions using the model/pipelines configurations tried during the optimization. Based on the intuition that diversity among the base models improves the quality of any ensemble, this paper seeks to modify a CASH solver to generate a more diverse set of base models, while still maintaining the predictive performance of the base models. To this end, the paper presents a notion of diversity between any two pair of model configurations, and shows how it can be incorporated into a Bayesian Optimization framework (BO) by proposing a new acquisition function that combines the predictive performance and diversity in a way that the predictive performance of selected models is maintained while improving the diversity among the base models. Empirically, the paper demonstrates the ability of the proposed DivBO scheme to generate diverse set of base models (based on their proposed definition of diversity) while solving the CASH problem. The empirical results also show that ensembles created by DivBO improve upon ensembles created by diversity agnostic CASH solvers.


strengths_and_weaknesses: ## Strengths

- The need for diversity in ensembles have been considered for a long time. It is great to see a CASH solver like DivBO that directly incorporates the desired diversity in the final ensemble within the CASH optimization instead of keeping the optimization and the ensembling completely separate as with most existing AutoML solutions that use CASH solvers. This is strong novel contribution.

- Given that diversity and predictive performance is not always aligned, the authors do a great job at clearly highlighting the challenges of incorporating diversity in a CASH solution and the intuitions guiding their proposed scheme. It is true that (i) there is no need to have high diversity from base models that do not perform well (and hence wont probably be part of the final ensemble), and (ii) high diversity should not be at the cost of predictive performance, since that form of diversity will not be useful for the final ensemble. To this end, the authors define an acquisition function for diversity that only relies on diversity from a pool of potential ensemble members, and is able to be incorporated with the acquisition function for the predictive performance with use of ranks instead of raw values.

- Another original strong idea in this paper is the way the predictive performance rank and the diversity rank are weighed in the final acquisition function (equation (5)). The use of a weight schedule is well motivated by the authors with the intuition that initially we want the CASH solver to focus on predictive performance (which allows us to get a strong pool of potential ensemble base learners), and gradually push the optimization to increase diversity without hurting the predictive performance too much.


- The authors provide a very concise but appropriate literature review.


## Weaknesses

- While it has been known that diversity in the base models of the ensemble can improve generalization, one key component is the definition of diversity. There is not much discussion on why the definition of diversity in equation (3) leads to improved ensemble performance (beyond the presented empirical results). As a reader, I would like a better motivation and understanding for this diversity metric since it is a critical part of this paper.

- Another point that is not clear is that, given that we are performing ensemble selection using the greedy selection scheme of Caruana et al (2004), would diversity play any role at all in the final ensemble selection? Given that we greedily select the next ensemble member which most improves the validation loss, what are conditions under which the increased diversity in base learners would lead to selection of the more diverse base learner compared to less diverse ones?

- From the results in Figure 3, it is not clear that DivBO is significantly better than RB-ES and BO-ES. Firstly, with so many baselines, it is more useful to consider pairwise comparisons (as seen in citation [24] and [A]) especially between closely performing baselines. Secondly, comparing the average rank without confidence intervals (or better, some form of Wilcoxon signed rank test of significance), it is not clear if DivBO improves over BO-ES and other baselines significantly because of the incorporated diversity. In fact, in Figure 4 and Table 3, the average performance improvement of DivBO against baselines is well within the predictive performance confidence intervals posted, undermining the expected improved generalization ability of an ensemble with diverse base models.



[A] Rakotoarison, Herilalaina, Marc Schoenauer, and Michèle Sebag. "Automated Machine Learning with Monte-Carlo Tree Search." IJCAI-19-28th International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization, 2019.


- Figure 5 (and also Figure 1) shows how different the next suggested configuration is to the ones in the current pool $\mathcal{P}$. Hence this highlights the ability of the combined acquisition function and its maximization (probably after the weight saturation) to propose diverse candidate configurations. However, it is not clear (i) how good is the predictive performance of these diverse configuration, (ii) how this improves the predictive performance of the ensemble. The authors also present the average validation performance of the base models in the ensemble pool in Figure 5, highlighting that the base models' performances do not drop because of the increased diversity, but it does not say anything about improving the ensemble performance -- it is quite possible that DivBO proposes diverse configurations but they are not included in the ensemble, in which case, neither the ensemble performance nor the average base model performance would be affected by the diversity. This figure does not seem to make it clear that the increased diversity is playing a significant role in improving the ensemble quality.


questions: - While DivBO- is considered in the experiments, are there results (empirical or theoretical) highlighting the difference between DivBO and DivBO without the weight schedule in equation (5); that is, the weight is fixed to $\beta$ ($w = \beta$)? While it seems intuitive that DivBO with the weight schedule would perform better, it would be good to have empirical evidence.

- The paper spends a lot of space on how the surrogate model for the diversity metric is created and how it generalizes. This is useful detail but lot of it appears to be straightforward given observations of the metric much like most surrogate modeling. It would great to understand what were the challenges in the surrogate modeling of the diversity which motivated the authors to describe and study the surrogate modeling of the diversity metric in such detail, instead of studying, for example, the effect of the different choices in DivBO (such as effect of $\beta$ and $\tau$ in the weight schedule in the main paper, or the number of samples for the combined acquisition function optimization).

- The definition of the diversity acquisition function is not entirely clear. It appears from the definition and the following text that, in one trial, for each of the $\theta \in \mathcal{P}$, we sample from $M_{div}(\theta, x)$ and then take a minimum over all $\theta \in \mathcal{P}$, and take an average over $N$ such trials. Can you please explain why we are using a mean over minimum-of-per-theta samples and not just minimum of per-theta-sample-means? This choice seems a bit unintuitive and I would like to understand if this plays a significant role in DivBO, and if so, why?


- Is BO exactly the same as DivBO- where $w=0$ throughout the optimization (or equivalently BO-ES and DivBO)? Otherwise different performance surrogate function and/or acquisition function maximization strategies will lead to differences and it won't be clear if the difference between BO and DivBO is because of the diversity incorporation or just because of other differences in the BO configurations.

- In the evaluation in section 4.2, the correlation between the true diversity and predicted diversity with $M_{div}$ appears positive but still is only around 0.5 even with 250 configurations implying  $250^2$ observations to fit the $M_{div}$ surrogate model, which seems to be a large number of observations (albeit non-i.i.d. ones). Is there an explanation for why the $M_{div}$ quality is not much higher? What are the potential challenges?

- In the definition of diversity in equation (3), the predicted class probability $T_{x_i}(s)$ for some configuration $x_i$ and validation sample $s$ require that the trained model generates probabilities. How are the class probabilities generated from discriminative models such as SVC (linear or kernel) which do not inherently produce class probabilities but rather directly output class labels?


- In Figure 6, why do the different schemes have different number of base learners? It is weird to compare different baselines with different number of ensembles.

limitations: The authors claim in the Checklist that they discuss the limitations of the proposed scheme in Figure 3 and Section 4.3, it is not clear from the figure or the subsection what limitations of DivBO they have discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this work, the authors aim to improve CASH solutions by favoring diverse predictors during the optimization which are more appropriate during post-hoc ensembling. In order to achieve this, the authors adapt the acquisition function in the Bayesian optimization by a diversity term. Therefore, selected candidates which are a trade-off between predictive performance and a diverse addition to the current ensemble. The authors compare their method against standard CASH methods, automated ensemble learning methods and AutoML methods + posthoc ensembling on 15 datasets.
strengths_and_weaknesses: **Summary**

The idea is reasonable, is of academic relevance and to some degree relevant for practitioners. The method is clearly described. The empirical evaluation raises some concerns regarding the usefulness of the proposed method. While the method usually ranks best, the improvement over a random search with post-hoc ensembling does not seem to be statistical significant and no significance test is provided to show otherwise. Further ablation studies could have been conducted to provided deeper understanding about the importance of diverse models and the hyperparameter sensitivity.

**Details**

It is common practice to select upfront a diverse set of algorithms, train a couple of them with different hyperparameters and use a weighted average across them. This raises the question how the proposed method compares against this simple baseline. A method in the literature that makes use of this fact is [1]: They decompose the CASH problem in K HPO problems, solve those and then use post-hoc ensembling. This will also result in diverse, well-performing models, mimics the human strategy but does not explicitly consider the search for diverse candidates given an ensemble. It would be interesting to understand whether this is important and whether the proposed method actually makes efficient use of it. At the moment we only know that the considered baseline methods are skewed towards better performing algorithms which reduces diversity. The solution in the paper mentioned above seems to be a very simple solution to overcome this problem.

Figure 6 is great since it demonstrates that DivBO is indeed finding ensembles with higher diversity. However, this is not particularly surprising given that the pool of candidates for the post-hoc ensembling is more diverse from the beginning. I think it would be more interesting to understand the importance of the diverse models for the ensemble. This could be done by using a greedy post-hoc ensembling approach and report the order of models or report the performance of the ensembling when removing the most diverse models. Optimally, those results are reported across all datasets and not a single one.

While the test error in Table 3 of DivBO is oftentimes the lowest, the improvement over BO-ES or even RS-ES is small and given the high standard deviation, I expect the improvement to not be statistically significant. It would be great if the authors could provide a significance test.

Addition of two new hyperparameters. The provided sensitivity analysis on a single dataset is insufficient to judge how drastically the optimal setting changes with different datasets. The authors mention that the sensitivity analysis was the motivation for the final choice of the hyperparameter settings. However, the optimal solution is at the edge of the grid. How does the grid look like for larger or smaller taus and larger betas? Getting an idea about the sensitivity based on only one dataset is hard. It would be important to understand whether the best settings are significantly different across datasets.


The use of an ensemble in the first place is a limitation which is not discussed. In practice, ensembles may be well-performing but they are typically not used since they are very hard to understand, have higher inference time and are harder to maintain.

**References**

[1] Martin Wistuba, Nicolas Schilling, Lars Schmidt-Thieme: Automatic Frankensteining: Creating Complex Ensembles Autonomously. SDM 2017: 741-749

questions: Is the improvement statistically significant?

How important are the most diverse models in the final ensemble?
limitations: Discussion on problems of ensemble models could be added.

Maybe the authors can be more precise about where they describe their limitations. The section referred to in the checklist does not seem to discuss limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposed a method to improve the ensemble performance under the CASH framework. The basis of the method is based on observations by many previous work that a diverse set of base learners will usually improve ensemble performance. The method jointly searches for configurations not only with a good objective but also diverse ones compared to the configurations that have been evaluated.

To achieve that, the paper proposed a surrogate to measure diversity between a pair of configurations and combine the acquisitions for quality and diversity through a weighted ranking. Experiments demonstrated improved performance compared to well thoughted baselines.

strengths_and_weaknesses: The strengths of the paper include:
- A novel idea to consider both diversity and performance during CASH. 
- Very good presentation quality with clear structure and detail.
- Informative experiments result to support the claims.

The weakness of the paper include:
- Since this is an empirical paper, I would expect evaluations on more datasets, especially given all the datasets are less than 20k rows. For example the [OpenML benchmark suites](https://openreview.net/forum?id=OCrD8ycKjG).

Nice to have:
- There is a strong baseline called [AutoGluon Tabular](https://github.com/awslabs/autogluon), which reports competitive empirical results by training with default HPs and then doing multi-layer stacking. It would be nice to know how the proposed method compares to it.

==== After reading the author's response ===

The authors addressed all my questions and comments, I will increase the score to 7.
questions: - I am not sure about “... while ensuring the performance of base learners” in line 328. Is the average performance a good measure to claim that? As we have seen in Figure 3 and the author also wrote in line 286, “We also find that DivBO- performs worse than BO without ensemble learning”. So the best base learners found by DivBO are clearly worse than others.
- Minor: In Figure 5, “-d” and “-p” are not explained.

limitations: - I would suggest the authors highlight the potential overfitting issue, which has been observed in their experiments.
- Also, the ensemble can lead to slow inference and one should still consider a single model if inference time is critical.
review_title: null

Meta-review: metareview: After a thorough discussion with the authors, all reviewers agree that the paper should be accepted at NeurIPS. The reviewers appreciated the idea of incorporating diversity in the combined algorithm selection and hyper-parameter optimization (CASH) framework and the subsequent use of the diverse models in an ensemble to improve performance. The paper is very clearly written, and the experimental evaluation shows that the proposed techniques provides small but consistent improvements in performance. The authors provided a comprehensive response where they addressed most of the reviewers' concerns. I expect the authors will incorporate all the new results in the camera ready version of the paper. 
confidence: Certain
Decision: Accept