DivBO: Diversity-aware CASH for Ensemble Learning
Authors: Yu Shen
Yupeng Lu
Yang Li
Yaofeng Tu
Wentao Zhang
Bin CUI
Venue: NeurIPS 2022 Accept (2022)
Keywords: Automated Machine Learning
Hyperparameter Optimization
CASH
Ensemble Learning
Diversity

Abstract: The Combined Algorithm Selection and Hyperparameters optimization (CASH) problem is one of the fundamental problems in Automated Machine Learning (AutoML). Motivated by the success of ensemble learning, recent AutoML systems build post-hoc ensembles to output the final predictions instead of using the best single learner. However, while most CASH methods focus on searching for a single learner with the best performance, they neglect the diversity among base learners (i.e., they may suggest similar configurations to previously evaluated ones), which is also a crucial consideration when building an ensemble. To tackle this issue and further enhance the ensemble performance, we propose DivBO, a diversity-aware framework to inject explicit search of diversity into the CASH problems. In the framework, we propose to use a diversity surrogate to predict the pair-wise diversity of two unseen configurations. Furthermore, we introduce a temporary pool and a weighted acquisition function to guide the search of both performance and diversity based on Bayesian optimization. Empirical results on 15 public datasets show that DivBO achieves the best average ranks (1.82 and 1.73) on both validation and test errors among 10 compared methods, including post-hoc designs in recent AutoML systems and state-of-the-art baselines for ensemble learning on CASH problems.

DivBO: Diversity-aware CASH for Ensemble Learning
Yu Shen', Yupeng Lu', Yang Li', Yaofeng Tu², Wentao Zhang Bin Cui!?
Keey Lab of High Confidence Software Technologies, Peking University, China Institute of Computational Social Science, Peking University (Qingdao), China
3 ZTE Corporation, China Data Platform, TEG, Tencent Inc., China
Mila - Québec Al Institute HEC, Montréal, Canada shenyu, xinkelyp. bin.cui @pku.edu.cn, tu. yaofeng @ zte com.cn Sthomasyngli ceiimaasyngli@teneeee tenccett.con "wentao.zhang @ mila.quebec
Abstract
The Combined Algorithm Selection and Hyperparameters optimization (CASH) problem is one of the fundamental problems in Automated Machine Learning (AutoML). Motivated by the success of ensemble learning, recent AutoML systems build post-hoc ensembles to output the final predictions instead of using the best single learner. However, while most CASH methods focus searching for single learner with the best performance, they neglect the diversity among base learners (i.c., they may suggest similar configurations to previously evaluated ones), which is also crucial consideration when building an ensemble. To tackle this issue and further enhance the ensemble performance, we propose DivBO, diversity-aware framework to inject explicit search of diversity into the CASH problems. In the framework, we propose use a diversity surrogate to predict the pair- wise diversity of two unseen configurations. Furthermore, we introduce temporary pool and a weighted acquisition function to guide the search of both performance and diversity based on Bayesian optimization. Empirical results on 15 public datasets show that DivBO achieves the best average ranks (1.82 and 1.73) on both validation and test emror among 10 compared methods, including post-hoc designs in recent AutoML systems and state-of-the-ee baselines for ensemble learning on CASH problems.
Introduction
In recent years, machine learning has made great strides in various application areas, e.g., computer recommendation [36, 37,, etc. However, it's often knowledge-intens intensive to develop customized solutions with promising performance, as the process includes selecting proper ML algorithms and tuning the hyperparameters, To reduce the barrier and facilitate the deployment of machine learning applications, the AutoML community raises the Combined Algorithm Selection and Hyperparameters optimization (CASH) problem to automate the optimization.
vision 14 [2]
39 and proposes several methods 33
18 15
for CASH focus on searching for the optimal performance of single learner, it's widely acknowledged that ensembles of promising learners often outperform single ones [44]] 3. For example, He et al. [14 won first place in ILSRVC 2015 with an average of several learners. And ensemble strategies can be frequently found in the top solutions of Kaggle com- petitions 16 4 Motivated by those achievements, recent AutoMI. systems (e.g., Auto-sklearn []]
While most AutoML methods B5
Auto- Auto-Pytorch 50 VolcanoMI. 25 build post-hoc ensembles based on all base learners from the
entire optimization process and show better empirical results than using the best single learner.
Despite the effectiveness of those post-hoc ensemble designs, the target of CASH methods is inconsistent with that of ensemble learning. In other words, a good ensemble should contain a pool of
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

base learners that are both well- vell-performing and diverse with each other 43 44 while most CASH methods only aim at searching for the berr-perfo configuration of learners. Figure shows the diversity of the suggested learners from Auto-sklearn and VolcanoMI. after 200 iterations. We define the wair-wise diversity of two learners as the average regularized Fuclidean distance of their probabilistic predictions which ranges from 0 to 1 (See Equation [3. And we show the minimum diversity of a suggested learner with other learners from the ensemble built at the 200 iteration. While most diversity values of the two AutoML systems are around 0.05, they suggest learners whose predictions are often quite similar to one of the learners obtained from previous iterations. It implies lack of diversity in previous post-hoe ensemble designs, and it also indicates that the performance of post-hoc ensembles might be further improved diversity is also taken into consideration when suggesting new configurations to evaluate. However, the search of diversity is non-trivial. A simple diversity-inducin algorithm would degenerate the pool of learners, e.g., diversity can be easily increased by predicting all samples for the classes that wrong and different from previous learners. Therefore, how to guide the search for both performance and diversity simultaneously in recent CASH algorithms is still open question.
In this paper, we propose DivBO, framework that combines Bayesian optimization (BO) with an explicit search of diversity for classi- fication. The contributions are summarized as:
algorithm
0:35
VolcanoML
DMBO
Minimum Diversity
0:30
Auto skieam
0:25
0.20
To the best of our knowledge, this work the first to enhance the ensemble performance of CASH methods by considering the diversity of suggested configurations.
0.16
230
200
290
240
250
2. To inject the search of diversity into CASH prob- lems, we design 1) the diversity surrogate to predict the piir-wise diversity given two unseen configura- tions and 2) BO-based framework that automati- cally balances the predictive performance and diversity of suggestions.
Iteration
Figure 1: Minimum diversity of suggested learners after 200 iterations quake. (20 runs)
3. Empirical results show that the diversity surrogate achieves better correlation with the ground- truth results compared with the performance surrogate used BO. Compared with recent AutoML systems, DivBO suggests configurations that are significantly more diverse with those in previous iterations (the red line Figure T. In addition, results on 15 public datasets show that DivBO achieves the best average rank (1.82 and 1.73) on both validation and test errors among 10 methods, including post-hoc designs in recent AutoMI systems and state-of-the- methods for ensemble learning on CASH problems.
2 Preliminary and Related Work
In this section, we review the related work and introduce the preliminary for our proposed method.
Combined algorithm selection and hyperparameters (CASH), We first introduce the basic nota- tions for the CASH problem. Let A be the set of K candidate algorithms A {A' ..... AK), Each algorithm A' has its corresponding hyperparameter space A Given dataset D = [Dirain, D } of a learning problem, the CASH problem aims to minimize the validation metric by searching for the best joint configuration x* = (A", X*) trained on the training set, where A" is the best algorithm and X its corresponding hyperparameters. For brevity, we use T(a,A) to denote the learner constructed by the joint configuration (a, A) and trained on Dtrain- The optimization objective can be formulated as:
min C(I(u,)), Dwat).
aEA.NEA
The CASH problem is first introduced by Auto-WEKA [39 and solved by applying Bayesian
optimization (BO) [35 [7 on the entire search space. Each BO iteration follows three steps: I)
BO fits probabilistic surrogate model based observations D { (#1, .... (an vn-1)) 1.) in which v; is the 7th evaluated joint configuration, and y: is its corresponding observed performance; 2) BO selects the most promising configuration z, by maximizing the acquisition function to balance exploration and exploitation; 3) BO evaluates the configuration 2,, (i.c., train the learner and obtain

---

its validation performance), and augment the observations D. To avoid confusion, we refer to the surrogate of BO as the performance surrogate Mpery the following section.
In addition to BO, TPOT [3] proposes use genetic programming solve the CASH problem.
The ADMM-based method [29] decomposes the problem into sub-problems and solves them us- ing ADMM [5] Rising bandit 23 continuously eliminates badly performing algorithms during
optimization. FLAML [41] suggests configurations based on estimated costs of improvement.
Ensemble-oriented CASH. Different from the original CASH, ensemble-oriented CASH aims to find an ensemble of m learners that minimizes the validation metric L, which is:
min
C(Ensemble(((an.)) "...,,((,,.. (amm..m))) Am)) Dvot).
(2)
a1 E.A.A mm EAam
To solve the problem, ensemble optimization [22 directly optimizes the target based on BQ by using an ensemble pool with fixed size. This strategy suffers from instability due the risk of adding a bad configuration during optimization. Neural ensemble search (NES) 43 is proposed based the regularized evolutionary algorithm [34]. The local search design in NES works well on low-dimensional problems like neural architecture search, but may encounter a bottleneck when applied to solve high-dimensional CASH problems.
Rather than directly optimizing on Equation 2 recent AutoML systems (e.g., Auto-skleam
VL. [25 26) Auto-Pytorch (50) adopt a specific ensemble strategy on the observation history
after solving Equation [ which refer to the post-hoc ensemble designs. While the optimization target is inconsistent, as shown in Section [ there's 's still space to improve the performance of post-hoc ensembles. Inspired by the theoretical study that diversity can help improve the performance of the ensemble [21 3 13 48). we follow the post-hoc ensemble designs and focus on generating a promising pool of learners for the final ensemble strategy.
Ensemble selection. Ensemble strategies methods that combine the predictions given a pool of learners, which are orthogonal to the direction of our method (i.e., suggesting a diverse pool of leamerr).. Among different ensemble strategies (e.g.. Bagging (9. Boosting [9] 30), Stacking (6), adopt the ensemble selection (7]. which works empirically well with AutoML as shown in previous study [ 25 43). In short, ensemble selection starts from empty ensemble and iteratively adds models from the pool with replacement to maximize the ensemble validation performance (with uniform weights). The pseudo-code is provided in Appendix A.
3 Diversity-aware Bayesian optimization (DivBO)
In this section, we present DivBO - our proposed diversity CASH method for ensemble learning based on Bayesian optimization (BO). To inject the search of diversity into BO, will answer the following two questions: 1) how to measure diversity and model the diversity of two unseen configurations, and 2) how suggest configurations that are both well-performing and diverse with potential ones in the final ensemble.
3.1 Diversity Surrogate
The existing diversity measures can be generally divided into pair-wise and nonpair-wise measures. The nonpair-wise diversity [7 directly measures a set of learners the ensemble. While the ensemble may change during each iteration, it's difficult to model the diversity of an ensemble with a candidate configuration and multiple learners. Therefore, we use the pair-wise measures to simply learn the diversity of two given configurations. To this end, we follow the definition in previous research [44] that explicitly improves the diversity of neural networks and shows satisfactory empirical results. Let z; denote as the joint configuration of algorithm a; and hyperparameters A₃. The diversity between two configurations (Ir, 2,)) is the average Euclidean distance of class probabilities predicted on the validation set [4 which is:
Dio 23)) = V2
Σ sED.et
(T) (s)l22
[Daat]
where Dent is the validation set, Tz, is the learner corresponding to configuration x; and fitted on the training set, and (s;(s) (s) is the predictive class probability on sample s. Obviously, the relationship
3

---

between the diversity and configuration pair (Iz, 22) ) is also a black-box function. Therefore, inspired by Bayesian optimization, we apply another surrogate to model this relationship. This surrogate, namely the diversity surrogate, takes a configuration pair as input and outputs the predictive mean and variance of the pair-wise diversity.
Fitting. During each iteration, DivBO generates the training set for the diversity surrogate Maie by computing the diversity value of each pair of the observed configurations. Note that, (Iz, 2y) and (2,,zz) 4:) lead the same diversity value, but the training set should include both of them to ensure symmetry. This leads about 2D)2 training samples, where [D] is the number of observations. To avoid fetching validation predictions by retraining, we store those predictions for all observations during optimization, which is a common trick applied in previous methods
22 43
Implementation. Concretely, the diversity surrogate Mase in DivBO is ensemble of several LightGBM models [19]. This implementation has the following two advantages: 1) The diversity surrogate gives predictions with uncertainty balance exploration and exploitation. Concretely, the predictive mean and variance of the surrogate is obtained by computing the mean and variance of outputs generated by different LightGBM models. 2) The time complexity of fitting the surrogate is relatively low. The cost of training a LightGBM model is D log DD, which is lower than O(DD)s) when fitting& a Gaussian Process as the performance surrogate. In Section 4 we will further compare different implementations of diversity surrogates on their ability to fit the relationship.
3.2 Diversity-aware Framework
Based on the diversity surrogate, we propose diversity-awan framework suggest configurations that are both well-performing and diverse. Before stepping into the design, consider straightforward strategy that we choose the most diverse configuration with the observation history during each iteration. This simple strategy has two obvious drawbacks: 1) First, there is no need to suggest configurations that are diverse from badly-performing ones in the observation history-we only expect the promising base learners in the final ensemble to be diverse with each other; 2) The diversity can be easily increased by suggesting a learner that predicts all samples for the classes that are wrong and different from previous learners. In this extreme case, the learner can not help improve the performance of the final ensemble. Therefore, how control the diversity of the suggested configurations is non-trivial. In the following, we explain how DivBO tackles the two drawbacks.
Suggesting diverse configurations. To tackle the first drawback, since which learners will appear in the final ensemble is unknown during optimization, DivBO proposes to use a temporary pool P to collect base learners that will probably be selected into the final ensemble. Concretely, the pool is built by applying ensemble selection to the observation history. When the optimization ends, this temporary pool is exactly the output of a post-hoc ensemble strategy (e.g., auto-sklearn (II) that ends at the previous iteration, and thus the configurations in the temporary pool can be regarded as the potential ones in the final ensemble. Then, we define the diversity acquisition function O'dio of each unseen configuration x as its predicted diversity value with the most similar configuration the temporary pool:
N
1 N
Oldiv (x) =
min Ma. PEP
(4)
where P is the temporary configuration pool and Man is the diversity surrogate. M th sampled value from the output distribution of Maic given the pair (0, rr,, and the final acquisition function is the average of N minimums via sampling. Note that, using the minimum diversity with learners in the temporary pool is more appropriate than the mean diversity. In extreme cases, when the algorithm may suggest a useless configuration that is exactly the same as a previously evaluated one, the minimum diversity is penalized to zero while the mean diversity is still larger than zero. By maximizing Equation 4 DivBO is able to suggest configurations that are diverse from other potential ones in the final ensemble during optimization.
di (0, x)) is the
Combining predictive performance and diversity. As mentioned in the second drawback, optimiz- ing diversity alone degenerates the predictive performance of the suggested configurations. DivBO tackles this issue by combining both the performance and diversity acquisition functions with saturating weight. Since the performance and diversity acquisition values are of different scales, we propose to use the sum of ranking values instead of directly adding the output values. During each

---

Algorithm 1: Algorithm framework of DivBO.
Input: The search budget B, the architecture search space X' the ensemble size E, the training and validation set Derains Dvat-
Initialize observations as D Ø;
2 while budget B does not exhaust do
[ < 5 then
Suggest a random configuration ā X;;
else
Fit performance surrogate Mpers and diversity surtogate Maic based on observations D; Build a temporary pool configurations as
P {0,,-.. ...; } EnsembleSSlectionn D. Duat, E);;
Compute the ranks of sampled configurations Rper and Raso based on the performance
and diversity surrogates Mperf, Maie and the temporary pool P;
Suggest a configuration = arg min (xx) based on Equation
=EX
10 Build and train the learner /s on D Dtrain and evaluate its performance on Drat as y
Update the observations D D {{;,ū)};
12 Generate pool of base configurations {01 OF} EnsembleSelection(] D Deats E3;
13
return the final ensemble Ensemble(To,, Toτ)
BO iteration, we sample sufficient configurations by random sampling from the entire space and local sampling on well-performed observed configurations [17]. Then calculate the performance and diversity acquisition value for each sampled configuration. Based on these values, we further rank the sampled configurations and obtain the ranking value of as ( and Raio (2:) for the performance and diversity acquisition function, respectively. Finally, given a configuration T, we define the weighted acquisition function a for DivBO as:
a(x:)) = R (z;) + w Rade
[(sigmoid(rt) rt) - -0.5,,
(5)
where w is the weight for the diversity acquisition, is the current number of BO iterations, and are two hyperparameters. To match the intuition that the optimization should focus on performance in the beginning and gradually shift its attention to diversity, DivBO applies saturating weight, where ranges from [0, 3) and controls the speed of approaching saturation. The goal of DivBO is to suggest configurations that minimize the function in Equation S
Algorithm illustrates the procedure of DivBO. During each iteration after initialization, DivBO fits the performance and diversity surrogates based on observations (Line 6); 2) builds temporary configuration pool by applying ensemble selection on the observation history (Line 7): 3) samples candidate configurations and compute their ranking values (Line 8); 4) suggests a configuration
that minimizes the combined ranking value Equation 5 (Line 9): 5) evaluates the suggested configuration on the validation set and then updates the observations (Lines 10-11).
3.3 Discussion
In this section, we provide the discussion on DivBO as follows:
Time complexity. As mentioned in Section 3.1 the time complexity of fitting the diversity surrogate is O(ID [D), and the complexity of building temporary pool is O(DD..ill))) where |Doatl the number of validation data samples, and [D] is the number of observations. For large datasets, we can prepare constant subset of validation samples for ensemble selection. Therefore, the time complexity depends on the choice of performance surrogate in BO. Concretely, the complexity of DivBO for each iteration is (()D)s) when using the Gaussian Process [35] and O(ID log DD) when using the probabilistic random forest [17]
Extension. As an abstract algorithm framework, the components of DivBO can be replaced to meet different requirements. Though this paper focuses CASH problems for classification, DivBO can also be applied to regression problems by defining a new diversity function based on regression predictions instead of Equation 3 In addition, DivBO is independent of the choice of
5

---

performance surrogate for Bayesian optimization, i.e., the performance surrogate can be replaced with state-of-the-ars ones proposed for specific scenarios.
DivBO does not target at optimizing Equation 2 directly. The foundation of its effectiveness lies in the claim that it is beneficial for ensembles not only have promising base learners, but also more diversity in their predictions.
Foundation. Like the diversity-driven methods in other scenarios 44 32).
The claim has been studied by extensive theoretical work [21] 3 13 48 20], and please refer to previous work for more details. In the following section, we will empirically show that the ensemble generated by DivBO outperforms the state-of- of-the-arttttt- art methods in real- CASH problems.
Limitations. The use of ensemble leads to higher inference latency than using the single best learner (approximately K times where K is the number of learners in the ensemble). This latency can be reduced with the aid of parallel computing if we have sufficient computational resources; As ensemble selection is fitted on the validation set, there's risk of overfitting on the test set for small datasets; DivBO using Equation 3 as the diversity function can not directly support algorithms that only predict class labels (e.g., SVC). Though Bi still works by converting the outputs class probability (like [1, 0, ....)), other diversity functions can be developed to support those algorithms better.
4 Experiments
In this section, we evaluate our proposed method on real-world CASH problems using public datasets. In the following, we list three main insights that we will investigate: 1) The diversity surrogate in DivBO can predict the diversity value of two unseen configurations well. 2) The DivBO framework outperforms the post-hoc designs used in recent AutoML systems and other competitive baselines for ensemble learning, in terms of both validation and test performance. 3) The base learners in the ensemble given by DivBO show similar average performance but enjoy higher diversity than those from other post-hoe designs.
4.1 Experiment Setup
Baselines. We compare the proposed DivBO with the following eight baselines - Three CASH
methods: 1) Random search (RS) M: 2) Bayesian optimization (BO) [17] 3) Rising Bandit (RB) 23
Two AutoML methods proposed for ensemble learning: 4) Ensemble optimization (EO) 22] 5)
Neural ensemble search (NES) 43); Three post-hoc designs: 6) Random search with post- hoc
ensemble (RS-ES): 7) Bayesian optimization with post-hoc ensemble (BO-ES) (T]] the default
strategy in Auto-sklcarn; 8) Rising bandit with post-hoc ensemble (RB-ES) 25]: the default strategy in VolcanoML. While for DivBO, we also implement the variant without post-hoe ensemble, which we denote as "DivBO.".
Datasets and search space. While recent AutoML systems differ in both search space and algorithm, to make a fair comparison of algorithms, we conduct the experiments on the same search space. Concretely, we slightly modify the search space of the AutoML system VolcanoML (25). The search space contains 100 hyperparameters in total, and the details of algorithms and feature engineering hyperparameters are provided in Appendix A.3. In addition, we use 15 public classification datasets that are collected from OpenML [40. whose number of samples ranges from 2k to 20k. More details about the datasets are provided in Appendix A.2.
Basic settings. Each dataset is split into three sets, which are the training (60%), validation (20%), and test (20%) sets. To evaluate the diversity surrogate, we use the Kendall-tau rank correlation as the metric since we only care about the ranking relationship between two pairs during optimization. For comparison with other baselines on CASH problems, we report the best-observed validation error during optimization and the final test error. While it takes a different amount of time to evaluate the same configuration on different datasets, we use the evaluation iterations as the unit of budget. Following VolcanoML [25] where each baseline evaluates approximately 250 configurations, we set the number of maximum iterations to 250. The evaluation of each method on each dataset is repeated 10 times, and we report the ttd. result by default. The hyperparameters 3 and T are set to 0.05 and 0.2 in DivBO. We provide more implementation details for other baselines and the sensitivity analysis in Appendix A.4 and A.5, respectively.

---

0.0
0.5
Kendall Tou
Kendall Tau
055
0.4
0.1
10
Number Observations
Number Observations
quake
(b) wind
Figure 2: Kendall Tau correlation of different diversity surrogates with standard deviations shaded. '-p' refers to the performance surrogate while the other three are diversity surrogates.
10
Average Validation Rank
R5
BO
DivBO-
6
EO
NES
RS-ES
4
BO-ES
RB-ES
DivBO
0
50
100
150
200
250
Number Observations
Figure 3: Average validation rank of 10 methods during optimization across 15 datasets. Note that ranks are relative measure of performance (the rank of all methods add up to 55), and an improvement in validation error in one method may influence the rank of another one.
4.2 Evaluation of Diversity Surrogate
To show the soundness of DivBO, we first provide an analysis of the fitting capability of different diversity surrogates during optimization. We take LightGBM (LGB) [9]. XGBoost (XGB) 8 and probabilistic random forest (PRF) as the candidates for diversity surrogate, where all three tree-based alternatives share the same time complexity ( log We evaluate 300 randomly chosen configurations, among which up to 250 configurations are used to fit the surrogate, and the left 50 are used for surrogate evaluation. Figure 2 shows the Kendall Tau correlation between the surrogate predictive means across different runs and the ground-truie results on two different datasets.
We observe that, in general, the correlation of the diversity surrogate improves as the observations increase. Among the three alternatives, XGBoost performs slightly better than PRF, and LightGBM performs better than the other two alternatives. Remarkably, LightGBM achieves strong Kendall Tau correlation of 0.65 and 0.58 on quake and wind when fitted with 250 observations, respectively. We also evaluate the performance surrogate (PRF-p) used in BO. Its correlation over the number of observations much lower than the diversity surrogate, and the correlation only 0.26 and 0.19 on quake and wind when fitted with 250 observations. The reason is that the search space for CASH problems is too large, and the performance surrogate can not fitted well using limited observations D. While the number of pair-wise samples is much larger than non-pair-wil samples (i.e., 2 vs. (D), the diversity surrogate captures more information from the observations. diversity surrogate can fit the diversity relationship between two configurations well and enjoys
result, the

---

Ruan
NES
Bas Obsrrnal
BO-ES
Dw80
Namber Obsevations
Namber of Obscraaiooss
Number Otservatoos
(a) Pollen
(b) Wind
(c) Average test rank
Figure 4: Figures (a) and (b): Best observed validation eπror with standard deviations shaded on two datasets. Figure (c); Average test rank of post-hoe ensemble designs across 15 datasets.
relatively strong correlation given limited budget. In the following, apply LightGBM as the diversity surrogate and evaluate DivBO on real-world AutoMI. tasks.
4.3 Evaluation of DivBO
Performance Analysis. In this section, we evaluate DivBO on 15 real-world CASH problems. Figure shows the rank of validation error of all compared methods during optimization. We get four observations from the figure: 1) Ensemble learning indeed helps improve the performance of CASH results. Though Rising Bandit (RB) is competitive, the rank of non-ensemble methods (RS, BO, DivBO-) at the 250th iteration is only 8.72, 7.21, and 8.22 among 10 compared methods, respectively; 2) Among methods with ensemble learning, the post-hoc designs outperform the other designs for ensemble learning (NES, EO). Note that, random search with ensemble selection (RS-ES) is a strong baseline. achieves a rank of 4.30 at the 250th iteration, which is better than that of NES (5.82) and EO (6.65). The reason is that the evolutionary algorithm in NES is not suitable for large search space while the performance of EO fluctuates optimization; 3) Among post-hoc designs, DivBO outperforms the other ones. While the second-best baseline achieves a rank of 2.58 at the t5 iteration, the rank of DivBO is 1.82, and consistently outperforms the other baselines after about 100 iterations. worse than BO without ensemble learning. The reason is that, the consideration of diversity is useless when searching for the single best classifier. But with the aid of the performance surrogate, DivBO- still performs better than random search. To demonstrate the validation eπrоr on specific datasets, we also plot the best-observed validation errors during optimization for six methods with ensemble learning in Figures 4(a) and [bb) Similar to the trend of rank, DivBO exceeds the second-bes, baseline RB-ES after about 100 iterations. We further compare the number required iterations for DivBO to achieve the same best validation errors as other methods. Concretely, DivBO achieves 1.52-1.544 and 1.67-2.53x speedups relative RB-ES and BO-ES on the two datasets. We provide ablation study on weight scheduling and comparison with other intuitive designs in Appendix A.5.
poor learner is added into its fixed ensemble during
We also find that DivBO- performs
Table demonstrates the test eπrогs and average rank on 15 datasets. We observe that the rank value of DivBO on error sometimes consistent with that on validation error. The reason that the distributions of the validation and test set are not exactly the same [8] In other words, the configuration with the best validation етrоr may not be the one with the best test error. But overall, post-hoc ensemble designs outperform other methods for ensemble learning, among which DivBO achieves the best test еггог on 10 of 15 datasets, and its average rank 1.73. The second best baseline is RB-ES, which performs the best on 3 out of 15 datasets, and its average rank is 2.90. The test rank post-hoe ensemble designs during optimization is also presented in Figure (cc)
To check whether the improvement of BivBO is statistically significant, we conduct the Wilcoxon signed-rank test on each dataset given two methods. The difference is significant when the value p S 0.05. We count the number of datasets if 1) the mean error DivBO is lower, and the difference is statistically significant (B); 2) the difference is not statistically significant (S); and 3) the mean error of DivBO is higher, and the difference statistically significant (W). The results are presented in Table 2 Though RB-ES is strong baseline, we observe that DivBO performs worse than RB-FS on 12 datasets and better on datasets. Appendix A.5, we provide additional experiments to study the effects of different ensemble strategies and ensemble sizes.

---

Table 1: Test error (%) with standard deviations and the average rank across different datasets.
Method
employee
bank32nh
<pu_aet
opu
ogg
elevators
house Nt.
CASH Methods
RS
5.32 = 0.08
18.43 + 0.79
6.17 =: 0.45
7.57 0.41
6.58 + 0.52
10:22 = 0.68
11.32 0.26
1.65 + 0.12
5,26 4: 0.06
18.46 4: 0.50
5.69 0.38
7.76 0.92
5.47 4 1.35
10.25 0.57
11.20 0.19
1.59 0.50
5.27 0.08
0.32
4.70 1.05
Methods for Ensemble Learning
BO
5.19 0.27
18:17 4: 0.58
3.85 0.02
7.12 0:45
3.54 + 0,76
10.34 = 0.61
11.14
1443 1:3:2:0 0.13
5.32 0.38
0.32
+
11.28
Post-hoc Designs
RS-ES
5.29 2: 0.15
18.30 1: 0.60
5.70 0.26
7.21 1: 0.30
1.45 1: 0.22
9.51 1: 0.28
11:21 0.38
1.39 2 0.15
BO-ES
5.25 + 0.15
18.41 +: 0.39
5.50 0.47
7.16 0.28
3.55 +: 0.78
9.61 0.36
11:06 0.33
1.35 4...0...
0.11
0.20
0.22
2.86
DivBO
5.16 d: 0.09
18:35 + 0.32
5.37 d: 0.23
7.04 +: 0.29
3.26 + 0.84
9.40 :: 0.28
10.80 1: 0.22
1.34 0.17
Mothod
pollen
puna321t
quake
satimage
spambase
wind
2dplanes
Average Rank
Methods
RS
51.53 0.50
10.50 +: 1.26
48.17 1.88
10.00 + 0.72
7.14
14.68 0.58
7.21 0.08
8.37
BO
49.64 4 3.39
10:42 d: 0.51
46.88 2.16
9.21 0.99
6.45 4.854
14.52 1: 0.00
7.15 0.06
6.47
11.21 0.39
47.98 1.56
0.94
6.71+1..00
14.11
7.20-
6.37
Methods for Ensemble Learning
EO
49.01
+: 10
9.74 1.60
46.58 d:: 1.48
9.42 1.11
6.41 0.66
14.62 à: 0.48
7.13 = 0.07
5.77
NES
51.36 1.53
10:63 0.54
46 1,15
6.23
14:20 0.50
0.08
5.00
Post-hoc Designs
RS-ES
19.69 1.63
10.58 +: 0.73
16.79 = 1.57
9.35 0.73
6.45 0.23
14.31 0.17
0.12
5.27
8O-ES
46.10
14:04
0.08
3:13
RB-ES
19.58 1: 1.34
7.85 0.43
16.70 1.34
8.55 1.36
6.12 0.38
13.98 1: 0.45
7.20 0.08
2.90
DivBO
49.25 1.35
8.07 0.90
45.55 1. 1.37
8.71 1.25
5.91 d. 0.45
13.93 3: 0.42
7.00 0.08
1.73
Table 2: Counts of datasets when DivBO performs statistically better (B), the same (S), and worse (W) than compared three baselines.
(a) vs. RS-ES
(b) vs. BO-ES
(c) vs. RB-ES
B
S
W
B
S
W
B S W
DivBO
13
0
DivBO
12
2
DivBO
3
Diversity Analysis. Finally, we analyze the optimization process of BO-ES, RB-ES, and DivBO. In T show the validation errors of learners during optimization. Without ensemble, the single learner suggested by DivBO- performs worse than BO and RB. Note that, this does not mean that DivBO suggests bad configurations. We randomly evaluate 300 configurations from the search space. The mean result of those diverse configurations is better than 88% of the random configurations. To show how the diversity during the search process affects the ensemble, we use the update times of the temporary pool as a metric and the results are shown in Table Since the temporary pool is built in the same way as the final ensemble, the temporary pool changes, the configuration suggested the previous iteration is included in the pool. Therefore, a change of the temporary pool at least indicates the suggested configuration affects the current ensemble. However, though the pool changes, the performance may not be improved due to the greedy mechanism, and thus we count the effective update times (i.e., the pool changes and the validation error of the ensemble decreases). updates very frequently in the beginning, we only calculate the mean effective update times of DivBO, RB-ES, and BO-ES on all datasets during the last 50 and 100 iterations. The pool relatively stable in the last 50 iterations, which also indicates budget of 250 iterations is sufficient for the datasets. We observe that, on average, DivBO will improve the temporary pool at least once the last 50 iterations. While the difference between BO-ES and DivBO is the diversity part, we attribute this frequency gain to the use of diversity during the search process.
the pool
In Figure 5 given the total budget of 250 evaluation iterations, we plot the minimum diversity the suggested learners (solid lines) after 200 iterations. The minimum diversity is defined as the diversity value (Equation 3) with the most similar learner in the ensemble built the previous iteration. The minimum diversity of suggestions given by BO-ES and RB-ES is around 0.05, which indicates that the configuration suggestion is similar to one of the configurations in the previous ensemble. While for DivBO, the diversity is around 0.23, which is much higher than those of BO-ES and RB-ES.

---

10
10
10
32
84
34
10
10
12
3.0
12
16
(a) RB-ES Disagreement: 0.13 (b) BO-ES Disagreement: 0.09
(c) DivBO Disagreement: 0.27
Figure 6: Pair-wise diversity of learners in the post-hoc ensemble on quake. The learners in each method are numbered in order of observation.
Erros
Table 3: Val errors (%) of single learners during optimization.
Mineium Diversity
0:30
50-15p
Dv60
0.45
BO-ES
RB-ES
DivBO
0:15
Average Bose
Val Erross (%)
44.03 + 2.58
43.85
2.49
44.33 + 2.67
0:10
-0.4
Table 4: Effective pool updates during optimization.
BO-ES
RB-ES
DivBO
Counts (last 100)
1.8
2.1
3.4
Figure 5: Minimum diversity of suggested learners (solid) and average error (dash) of base learners in the ensemble on quake.
Counts (last 50)
0.6
0.8
1.5
In addition, we plot the average validation error of base learners in the current ensemble (dash lines) in Figure The average error at the t50th iteration given by the three post-hoe methods are quite similar (around 44.60%) which shows that applying the diversity surrogate DivBO will not degenerate the pool of base learners. We further plot the pair-wise diversity between unique learners in the final ensemble Figure and and present the average pair-wise predictive disagreement [38] in the caption. Generally, the larger disagreement is, the more diverse the learners in the ensemble are. Since ensemble selection selects base learners with replacement, the number of unique learners the ensemble is different in each independent run. Though the average performance of base learners in three post-hoc methods are similar, for RB-ES and BO-ES, the base learners are quite similar to each other. While for DivBO, we find that the learners are more diverse. Specifically, the learners that are found later (with larger number) are more diverse with each other. The reason is that the diversity surrogate becomes more precise when fitted with more observations, and thus it can suggest more diverse configurations. This observation indicates that DivBO able to generate a more diverse ensemble while ensuring the performance of base learners.
Conclusion
In this paper, we introduced DivBO, a diversity-aware framework based on Bayesian optimization to solve CASH problems for ensemble learning. In DivBO, proposed to model the relationship between two configurations, and combined the ranking values of the performance and diversity surrogates with saturating weight. Through empirical study, we showed that the prediction of the diversity surrogate achieves a satisfactory correlation with ground-truth results, and DivBO outperforms post-hoc designs recent AutoML systems and other baselines for ensemble learning in CASH problems.
diversity surrogate
Acknowledgments
This work supported by NSFC (No. 61832001) and ZTE-PKU Joint Laboratory for Foundation Software. Yang Li and Bin Cui are the corresponding authors.
10

---

References
[1] J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 13(Feb):281-335, 2012.
[2] Bergstra, Bardenet, Y. Bengio, and B. Kégl. Algorithms for hlpee--arrameter optimiza-
tion. In Advances in neural information processing systems, pages 2546-2554, 2011.
[3] Y. Bian and H. Chen. When does diversity help generalization in classification ensembles? IEEE Transactions on Cybernetics, 2021.
[4] C. Bojer and J. P. Meldgaard. Kaggle forecasting competitions: An overlooked learning opportunity. International Journal of Forecasting, 37(2):587-600 2021.
[5] S. Boyd, N. Parikh, E. Chu, B. Peleato, J. Eckstein, et al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends@ in Machine learning, 3(1):1-122, 2011.
[6] L. Breiman. Stacked regressions. Machine learning, 24(1):49-64 1996.
[7] R. Caruana, A. Niculescu-Mine Mizil, G. Crew, and A. Ksikes. Ensemble selection from libraries of models. In Proceedings of the twenty-first international conference on Machine learning. page 18, 2004.
[8] T. Chen and C. Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794. ACM, 2016.
[9] G. Dietterich. Ensemble methods in machine learning. International workshop on multiple classifier systems, pages 1-15. Springer, 2000.
[10] G. FAN, B. LI. Q. HAN, R. JIAO, and G. QU. Robust lane detection and tracking based on machine vision. ZTE Communications, 18(4):69-77, 2021.
[11] M. Feurer, A. Klein, K. Eggensperger, J. Springenberg, M. Blum, and Hutter. Efficient and robust automated machine learning. In Advances in neural information processing systems, pages 2962-2970, 2015.
[12] I. Goodfellow, J. Pouget- Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[13] L. K. Hansen and P. Salamon. Neural network ensembles. IEEE transactions on pattern analysis and machine intelligence, 12(10):993-1001 1990.
[14] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[15] X. He, K. Zhao, and X. Chu. Automl: A survey of the state-of-the-art. Knowledge-Based Systems, 212: 1066226 2021.
[16] T. Hoch. An ensemble learning approach for the kaggle taxi travel time prediction challenge. In DC@ PKDD/ECML, 2015.
[17] Hutter, H. H. Hoos, and K. Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In International Conference on Learning and Intelligent Optimization, pages 507-523 Springer, 2011.
[18] Hutter, L.. Kotthoff, and Vanschoren. Automated machine learning: methods, systems, challenges. Springer Nature, 2019.
[19] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu. Lightghm: A highly efficient gradient boosting decision tree. Advances neural information processing systems, 30, 2017.

---

[20] L. I. Kuncheva and C. J. Whitaker. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Machine learning, 510))1811-07,.003. 2003.
[21] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature, 521 521(7553):3364444 2015.
[22] J.-C. Lévesque, C. Gagné, and R. Sabourin. Bayesian hyperparameter optimization for en- semble learning. In Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence, pages 337-446, 2016.
[23] Y. Li, J. Jiang, J. Gao, Y. Shao, C. Zhang, and B. Cui. Efficient automatic cash via rising bandits. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4763-4771, 2020.
[24] Y. Li, Y. Shen, J. Jiang, J. Gao, C. Zhang, and B. Cui. Mbes-hb: Efficient hyperband with multi-fidelity quality measurements. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 8491-8500, 2021.
[25] Y. Li, Y. Shen, W. Zhang, Jiang, B. Ding, Y. Li, Zhou, Z. Yang, W. Wu, C. Zhang, et al. Volcanoml: speeding up end-to-end automl via scalable search space decomposition. Proceedings of the VLDB Endowment, 14(11):116722176 2021.
[26] Y. Li, Y. Shen, W. Zhang, C. Zhang, and B. Cui. Efficient end-to-end automl via scalable search space decomposition. The VLDB Journal, 2022.
[27] S. Limin, Z. Qiang, L. Shuang, and L. C. Harold. Balanced discriminative transfer feature learning for visual domain adaptation. ZTE Communicationss 18(4):78- 83, 2021.
[28] C. Liu, B. P. Chamberlain, and E. J. McCoy. What is the value of experimentation and measurement? Data Science and Engineering, 5(2):152-167, 2020.
[29] Liu, P. Ram, Vijaykeerthy, D. Bouneffouf, Bramble, H. Samulowitz, D. Wang, A. Conn, and A. Gray. An admm based framework for automl pipeline configuration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4892-4899, 2020.
[30] M. Moghimi, S. J. Belongie, M. J. Saberian, J. Yang, N. Vasconcelos, and L.-J Li. Boosted convolutional neural networks. In BMVC, volume 5, page 6, 2016.
[31] R. S. Olson and J. H. Moore. Tpot: A tree-based pipeline optimization tool for automating machine learning. In Workshop on automatic machine learning, pages 66-74, PMLR, 2016.
[32] I. Partalas, G. Tsoumakas, and I. P. Vlahavas. Focused ensemble selection: A diversity- based method for greedy ensemble selection. In ECAI, pages 1 117-121, 2008
[33] Y. Quanming, W. Mengshuo, J. E. Hugo, G. Isabelle, H. Yi-Qi, L.. Yu-Feng, T. Wei- Wei-Wei, Y. Qiang, and Y. Yang. Taking human out of learning applications: A survey on automated 2018
machine learning. arXiv preprint 1810.
[34] F. Real, A. Aggarwal, Y. Huang, and Q V. Le. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 4780-4789 2019.
[35] Snock, H. Larochelle, and R. P. Adams. Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems, pages 2951-2955 2012.
[36] X. Su and T. M. Khoshgoftaar. A survey of collaborative filtering techniques. Advances in artificial intelligence, 2009, 2009.
[37] F. Sun, J. Liu, J. Wu, C. Pei, X. Lin, W. Ou, and P. Jiang. Bert4ree: Sequential recommendation with bidirectional encoder representations from transformer. In Proceedings of the 28th ACM international conference on information and knowledge management, pages 1441-1450, 2019.
[38] E. Tang, P. N. Suganthan, and Yao. An analysis of diversity measures. Machine learning, 65(1):247-271, 2006.
12

---

[39] C. Thornton, F. Hutter, H. H. Hoos, and K. Scyton-Brown Auto-weka: Combined selection and hyperparameter optimization of classification algorithms. Proceedings of the 19th ACM SIGKDD international conference Knowledge discovery and data mining, pages 847-855, 2013.
[40] J. Vanschoren, N. Van Rijn, B. Bischl, and L. Torgo. Openml: networked science machine learning. ACM SIGKDD Explorations Newsletter, 15(2):49 60, 2014.
[41] C. Wang, Q. Wu, M. Weimer, and E. Zhu. Flaml: a fast and lightweight automl library. Proceedings of Machine Learning and Systems, 3:434-447, 2021.
[42] L. Yang, Z. Zhang, and S. Hong. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint arXiv:2209.000 00796, 2022.
[43] S. Zaidi, A. Zela, T. Elsken, C. C. Holmes, F. Hutter, and Y. Teh. Neural ensemble search for uncertainty estimation and dataset shift. Advances in Neural Information Processing Systems, 34, 2021.
[44] W. Zhang. J. Jiang, Y. Shao, and B. Cui. Efficient diversity-driven ensemble for deep neural networks. In 2020 IEEE 36th International Conference on Data Engineering (ICDE), pages 73 84. IEEE, 2020.
[45] W. Zhang, J. Jiang, Y. Shao, and B. Cui. Snapshot boosting: a fast ensemble framework for deep neural networks. Science China Information Sciences, 63(1):1-12, 2020.
[46) W. Zhang, Shen, Z. Lin, Li, X. Li, W. Ouyang, Y. Tao, Z. Yang, and B. Cui. Pasca: graph neural architecture search system under the scalable paradigm. In Proceedings of the ACM Web Conference 2022, pages 1817-1828, 2022.
[47] Z.-H. Zhou. Ensemble methods: foundations and algorithms. CRC press, 2012.
[48] Z.-H. Zhou, J. Wu, and W. Tang. Ensembling neural networks: many could be better than all. Artificial intelligence, 137(1-2):33-266. 2002.
[49] D.-H. Zhu, X.-Y. Dai, and J.-J Chen. Pre-train and learn: Preserving global information for
graph neural networks. Journal of Computer Science and Technology, 36(6):1400-1430 2021.
[50] L. Zimmer, M. Lindauer, and F. Hutter. Auto-pytorch: multi-fidelity metalearning for efficient and robust autodl. IEEE Transactions Puttern Analysis and Machine Intelligence, 43(9):3079- 3090, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 3.3
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We have carefully read the guidelines.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We have a detailed implementation instruction in Appendix A.4.
13

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Table and Figure
(d) Did you include the total amount compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix A.4.
4. If you are using existing assets (e.g., code, data, models) or curating/releasnng new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] We use the widely used and public data OpenMI.. The datasets are used for evaluation in previous work.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "xGsbS6XKhBl", "reviewer": "Reviewer_t73G", "summary": "This paper focuses on the problem of CASH (Combined Algorithm Selection and Hyperparameter Optimization) to automatically configure machine learning models (or pipelines), and seeks to improve the quality of the final ensemble generated by various AutoML solutions using the model/pipelines configurations tried during the optimization. Based on the intuition that diversity among the base models improves the quality of any ensemble, this paper seeks to modify a CASH solver to generate a more diverse set of base models, while still maintaining the predictive performance of the base models. To this end, the paper presents a notion of diversity between any two pair of model configurations, and shows how it can be incorporated into a Bayesian Optimization framework (BO) by proposing a new acquisition function that combines the predictive performance and diversity in a way that the predictive performance of selected models is maintained while improving the diversity among the base models. Empirically, the paper demonstrates the ability of the proposed DivBO scheme to generate diverse set of base models (based on their proposed definition of diversity) while solving the CASH problem. The empirical results also show that ensembles created by DivBO improve upon ensembles created by diversity agnostic CASH solvers.\n\n", "questions": "- While DivBO- is considered in the experiments, are there results (empirical or theoretical) highlighting the difference between DivBO and DivBO without the weight schedule in equation (5); that is, the weight is fixed to $\\beta$ ($w = \\beta$)? While it seems intuitive that DivBO with the weight schedule would perform better, it would be good to have empirical evidence.\n\n- The paper spends a lot of space on how the surrogate model for the diversity metric is created and how it generalizes. This is useful detail but lot of it appears to be straightforward given observations of the metric much like most surrogate modeling. It would great to understand what were the challenges in the surrogate modeling of the diversity which motivated the authors to describe and study the surrogate modeling of the diversity metric in such detail, instead of studying, for example, the effect of the different choices in DivBO (such as effect of $\\beta$ and $\\tau$ in the weight schedule in the main paper, or the number of samples for the combined acquisition function optimization).\n\n- The definition of the diversity acquisition function is not entirely clear. It appears from the definition and the following text that, in one trial, for each of the $\\theta \\in \\mathcal{P}$, we sample from $M_{div}(\\theta, x)$ and then take a minimum over all $\\theta \\in \\mathcal{P}$, and take an average over $N$ such trials. Can you please explain why we are using a mean over minimum-of-per-theta samples and not just minimum of per-theta-sample-means? This choice seems a bit unintuitive and I would like to understand if this plays a significant role in DivBO, and if so, why?\n\n\n- Is BO exactly the same as DivBO- where $w=0$ throughout the optimization (or equivalently BO-ES and DivBO)? Otherwise different performance surrogate function and/or acquisition function maximization strategies will lead to differences and it won't be clear if the difference between BO and DivBO is because of the diversity incorporation or just because of other differences in the BO configurations.\n\n- In the evaluation in section 4.2, the correlation between the true diversity and predicted diversity with $M_{div}$ appears positive but still is only around 0.5 even with 250 configurations implying  $250^2$ observations to fit the $M_{div}$ surrogate model, which seems to be a large number of observations (albeit non-i.i.d. ones). Is there an explanation for why the $M_{div}$ quality is not much higher? What are the potential challenges?\n\n- In the definition of diversity in equation (3), the predicted class probability $T_{x_i}(s)$ for some configuration $x_i$ and validation sample $s$ require that the trained model generates probabilities. How are the class probabilities generated from discriminative models such as SVC (linear or kernel) which do not inherently produce class probabilities but rather directly output class labels?\n\n\n- In Figure 6, why do the different schemes have different number of base learners? It is weird to compare different baselines with different number of ensembles.\n", "limitations": "The authors claim in the Checklist that they discuss the limitations of the proposed scheme in Figure 3 and Section 4.3, it is not clear from the figure or the subsection what limitations of DivBO they have discussed. \n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The need for diversity in ensembles have been considered for a long time. It is great to see a CASH solver like DivBO that directly incorporates the desired diversity in the final ensemble within the CASH optimization instead of keeping the optimization and the ensembling completely separate as with most existing AutoML solutions that use CASH solvers. This is strong novel contribution.\n\n- Given that diversity and predictive performance is not always aligned, the authors do a great job at clearly highlighting the challenges of incorporating diversity in a CASH solution and the intuitions guiding their proposed scheme. It is true that (i) there is no need to have high diversity from base models that do not perform well (and hence wont probably be part of the final ensemble), and (ii) high diversity should not be at the cost of predictive performance, since that form of diversity will not be useful for the final ensemble. To this end, the authors define an acquisition function for diversity that only relies on diversity from a pool of potential ensemble members, and is able to be incorporated with the acquisition function for the predictive performance with use of ranks instead of raw values.\n\n- Another original strong idea in this paper is the way the predictive performance rank and the diversity rank are weighed in the final acquisition function (equation (5)). The use of a weight schedule is well motivated by the authors with the intuition that initially we want the CASH solver to focus on predictive performance (which allows us to get a strong pool of potential ensemble base learners), and gradually push the optimization to increase diversity without hurting the predictive performance too much.\n\n\n- The authors provide a very concise but appropriate literature review.\n\n\n## Weaknesses\n\n- While it has been known that diversity in the base models of the ensemble can improve generalization, one key component is the definition of diversity. There is not much discussion on why the definition of diversity in equation (3) leads to improved ensemble performance (beyond the presented empirical results). As a reader, I would like a better motivation and understanding for this diversity metric since it is a critical part of this paper.\n\n- Another point that is not clear is that, given that we are performing ensemble selection using the greedy selection scheme of Caruana et al (2004), would diversity play any role at all in the final ensemble selection? Given that we greedily select the next ensemble member which most improves the validation loss, what are conditions under which the increased diversity in base learners would lead to selection of the more diverse base learner compared to less diverse ones?\n\n- From the results in Figure 3, it is not clear that DivBO is significantly better than RB-ES and BO-ES. Firstly, with so many baselines, it is more useful to consider pairwise comparisons (as seen in citation [24] and [A]) especially between closely performing baselines. Secondly, comparing the average rank without confidence intervals (or better, some form of Wilcoxon signed rank test of significance), it is not clear if DivBO improves over BO-ES and other baselines significantly because of the incorporated diversity. In fact, in Figure 4 and Table 3, the average performance improvement of DivBO against baselines is well within the predictive performance confidence intervals posted, undermining the expected improved generalization ability of an ensemble with diverse base models.\n\n\n\n[A] Rakotoarison, Herilalaina, Marc Schoenauer, and Michèle Sebag. \"Automated Machine Learning with Monte-Carlo Tree Search.\" IJCAI-19-28th International Joint Conference on Artificial Intelligence. International Joint Conferences on Artificial Intelligence Organization, 2019.\n\n\n- Figure 5 (and also Figure 1) shows how different the next suggested configuration is to the ones in the current pool $\\mathcal{P}$. Hence this highlights the ability of the combined acquisition function and its maximization (probably after the weight saturation) to propose diverse candidate configurations. However, it is not clear (i) how good is the predictive performance of these diverse configuration, (ii) how this improves the predictive performance of the ensemble. The authors also present the average validation performance of the base models in the ensemble pool in Figure 5, highlighting that the base models' performances do not drop because of the increased diversity, but it does not say anything about improving the ensemble performance -- it is quite possible that DivBO proposes diverse configurations but they are not included in the ensemble, in which case, neither the ensemble performance nor the average base model performance would be affected by the diversity. This figure does not seem to make it clear that the increased diversity is playing a significant role in improving the ensemble quality.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RcP-nahbnPj", "reviewer": "Reviewer_15EU", "summary": "In this work, the authors aim to improve CASH solutions by favoring diverse predictors during the optimization which are more appropriate during post-hoc ensembling. In order to achieve this, the authors adapt the acquisition function in the Bayesian optimization by a diversity term. Therefore, selected candidates which are a trade-off between predictive performance and a diverse addition to the current ensemble. The authors compare their method against standard CASH methods, automated ensemble learning methods and AutoML methods + posthoc ensembling on 15 datasets.", "questions": "Is the improvement statistically significant?\n\nHow important are the most diverse models in the final ensemble?", "limitations": "Discussion on problems of ensemble models could be added.\n\nMaybe the authors can be more precise about where they describe their limitations. The section referred to in the checklist does not seem to discuss limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Summary**\n\nThe idea is reasonable, is of academic relevance and to some degree relevant for practitioners. The method is clearly described. The empirical evaluation raises some concerns regarding the usefulness of the proposed method. While the method usually ranks best, the improvement over a random search with post-hoc ensembling does not seem to be statistical significant and no significance test is provided to show otherwise. Further ablation studies could have been conducted to provided deeper understanding about the importance of diverse models and the hyperparameter sensitivity.\n\n**Details**\n\nIt is common practice to select upfront a diverse set of algorithms, train a couple of them with different hyperparameters and use a weighted average across them. This raises the question how the proposed method compares against this simple baseline. A method in the literature that makes use of this fact is [1]: They decompose the CASH problem in K HPO problems, solve those and then use post-hoc ensembling. This will also result in diverse, well-performing models, mimics the human strategy but does not explicitly consider the search for diverse candidates given an ensemble. It would be interesting to understand whether this is important and whether the proposed method actually makes efficient use of it. At the moment we only know that the considered baseline methods are skewed towards better performing algorithms which reduces diversity. The solution in the paper mentioned above seems to be a very simple solution to overcome this problem.\n\nFigure 6 is great since it demonstrates that DivBO is indeed finding ensembles with higher diversity. However, this is not particularly surprising given that the pool of candidates for the post-hoc ensembling is more diverse from the beginning. I think it would be more interesting to understand the importance of the diverse models for the ensemble. This could be done by using a greedy post-hoc ensembling approach and report the order of models or report the performance of the ensembling when removing the most diverse models. Optimally, those results are reported across all datasets and not a single one.\n\nWhile the test error in Table 3 of DivBO is oftentimes the lowest, the improvement over BO-ES or even RS-ES is small and given the high standard deviation, I expect the improvement to not be statistically significant. It would be great if the authors could provide a significance test.\n\nAddition of two new hyperparameters. The provided sensitivity analysis on a single dataset is insufficient to judge how drastically the optimal setting changes with different datasets. The authors mention that the sensitivity analysis was the motivation for the final choice of the hyperparameter settings. However, the optimal solution is at the edge of the grid. How does the grid look like for larger or smaller taus and larger betas? Getting an idea about the sensitivity based on only one dataset is hard. It would be important to understand whether the best settings are significantly different across datasets.\n\n\nThe use of an ensemble in the first place is a limitation which is not discussed. In practice, ensembles may be well-performing but they are typically not used since they are very hard to understand, have higher inference time and are harder to maintain.\n\n**References**\n\n[1] Martin Wistuba, Nicolas Schilling, Lars Schmidt-Thieme: Automatic Frankensteining: Creating Complex Ensembles Autonomously. SDM 2017: 741-749\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NdCLAPKMxW5", "reviewer": "Reviewer_anrp", "summary": "This work proposed a method to improve the ensemble performance under the CASH framework. The basis of the method is based on observations by many previous work that a diverse set of base learners will usually improve ensemble performance. The method jointly searches for configurations not only with a good objective but also diverse ones compared to the configurations that have been evaluated.\n\nTo achieve that, the paper proposed a surrogate to measure diversity between a pair of configurations and combine the acquisitions for quality and diversity through a weighted ranking. Experiments demonstrated improved performance compared to well thoughted baselines.\n", "questions": "- I am not sure about “... while ensuring the performance of base learners” in line 328. Is the average performance a good measure to claim that? As we have seen in Figure 3 and the author also wrote in line 286, “We also find that DivBO- performs worse than BO without ensemble learning”. So the best base learners found by DivBO are clearly worse than others.\n- Minor: In Figure 5, “-d” and “-p” are not explained.\n", "limitations": "- I would suggest the authors highlight the potential overfitting issue, which has been observed in their experiments.\n- Also, the ensemble can lead to slow inference and one should still consider a single model if inference time is critical.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths of the paper include:\n- A novel idea to consider both diversity and performance during CASH. \n- Very good presentation quality with clear structure and detail.\n- Informative experiments result to support the claims.\n\nThe weakness of the paper include:\n- Since this is an empirical paper, I would expect evaluations on more datasets, especially given all the datasets are less than 20k rows. For example the [OpenML benchmark suites](https://openreview.net/forum?id=OCrD8ycKjG).\n\nNice to have:\n- There is a strong baseline called [AutoGluon Tabular](https://github.com/awslabs/autogluon), which reports competitive empirical results by training with default HPs and then doing multi-layer stacking. It would be nice to know how the proposed method compares to it.\n\n==== After reading the author's response ===\n\nThe authors addressed all my questions and comments, I will increase the score to 7.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Dg_A3p0_NDx", "reviewer": "Reviewer_eDCC", "summary": "This works present a method for hyperparameter optimization applied to ensemble construction. The proposed method builds a surrogate function for diversity of pairs of classifiers that is then combined with a predictive performance objective and maximized. Experiments show that the method outperforms other ensemble building approaches in the literature.", "questions": "The paper and method description are clear and self-contained. The question is are the contributions significant enough to be published at Neurips. The performance improvement is small between DiVBO and RB-ES but the experiment compares a wide number of approaches and showcases RB-ES itself I believe for the first time. Find some questions/suggestions below. \n\n- I don't think showing the ranking of validation errors on Figure 3 is very informative, especially given the small size of some of the datasets -- overfitting is likely happening. You should show the ranking of testing error and push back the validation plot to the appendix. Same thing goes for Figure 4 (or maybe reduce number of methods displayed, prune the worse performing methods such as EO, NES and RS-ES and show both validation and testing errors). \n\n- lines 177-179: are the configurations sampled once per objective function? how many points are sampled?\n\n- line 235: we slightly modifies -> modify\n\n- I think section titles 4.2 and 4.3 should be evaluation OF DivBO/Diversity surrogate and not evaluation on?", "limitations": "There is a limited discussion of limitations of the method. There are no potential negative societal impacts warranting a discussion.\n", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The contributions are incremental, inscribed in the already existing framework of ensemble learning through hyperparameter optimization\n- The paper is well written and structured\n- The quality of the experiments is good\n\nWeaknesses:\n- The improvement in performance in comparison with the second best method RB-ES is hinting towards a plateau in performance in terms of ensembling\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept