Recycling sub-optimial Hyperparameter Optimization models to generate efficient Ensemble Deep Learning
Authors: Pierrick Pochelu
Bruno Conche
Serge G. Petiton
Venue:  (2021)
Keywords: Deep Learning
hyperparameter optimization
ensemble deep learning
multi-GPU

Abstract: Ensemble Deep Learning improves accuracy over a single model by combining predictions from multiple models.  It has established itself to be the core strategy for tackling the most difficult problems, like winning Kaggle challenges. Due to the lack of consensus to design a successful deep learning ensemble, we introduce Hyperband-Dijkstra, a new workflow that automatically explores neural network designs with Hyperband and efficiently combines them with Dijkstra's algorithm.  This workflow has the same training cost than standard Hyperband running except sub-optimal  solutions are stored  and  are  candidates  to  be  selected in  the  ensemble  selection  step (recycling). Next, to predict on new data, the user gives to Dijkstra the maximum number of models wanted in  the  ensemble to control the tradeoff between accuracy and inference time.  
Hyperband is a very efficient algorithm allocating exponentially more resources to the most promising configurations. It is also capable to propose diverse models due to its pure-exploration nature, which allows Dijkstra algorithm with a smart combination of diverse models to achieve a strong variance and bias reduction. The exploding number of possible combinations generated by Hyperband increases the probability that Dijkstra finds an accurate combination which fits the dataset and generalizes on new data.
The two experimentation on CIFAR100 and on our unbalanced microfossils dataset show that our new workflow generates an ensemble far more accurate than any other ensemble of any ResNet models from ResNet18 to ResNet152.

Under review as a conference paper at ICLR 2021
RECYCLING SUB-OPTIMIAL HYPERPARAMETER OPTI- MIZATION MODELS TO GENERATE EFFICIENT ENSEM- BLE DEEP LEARNING
Anonymous authors Paper under double-bling blind review
ABSTRACT
Ensemble Deep Learning improves accuracy single model by combining predictions from multiple models. has established itself to he the core strategy for tackling the most difficult problems, like winning Kaggle challenges. Due the lack of consensus to design a successful deep learning ensemble, we introduce Hyperband Dijjkstr,, a new workflow that automatically explores neural network designs with Hyperband and efficiently combines them with Dijkstra's 's algorithm. This workflow has the same training cost than standard Hyperband running except sub-optimal solutions are stored and are candidates to be selected in the ensemble selection step (recycling). Next, to predict on new data, the user gives to Dijkstra the maximum number of models wanted in the ensemble to control the tradeoff between accuracy and inference time. Hyperband is a very efficient algorithm al- locating exponentially more resources the most promising configurations. It is also capable to propose diverse models due to its pure-exploration nature, which allows Dijkstra algorithm with smart combination of diverse models to achieve a strong variance and bias reduction. The exploding number of possible combi- nations generated by Hyperband increases the probability that Dijkstra finds accurate combination which fits the dataset and generalizes on new data. The two experimentation on CIFAR100 and on our unbalanced microfossils dataset show that our new workflow generates an ensemble far more accurate than any other ensemble of any ResNet models from ResNet18 to ResNet152.
INTRODUCTION
Ensemble machine learning is popular method use predictions and combine them for success- ful and optimal classification.
In the light of its success in Kaggle competition, all top-5 solutions published in the last seven image recognition challenges use at least one ensemble method. The average and median number of individual models used by ensemble is between 7 and 8. Appendix(A Appeedix(Alsummariiedd these 17 solutions.
Despite its recent popularity among practitioners, there is no consensus on how to apply ensemble in the context of deep neural network. The overall work on ensemble Machine Learning (non-deep) was carried out in the 1990s and 2000s. The implementation of Deep Learning on GPU appeared less than years ago. The outbreak of multi-GPU servers allows to effectively train and evaluate many neural networks simultaneously but also deploy ensemble deep architectures.
Another recent trend to improve accuracy is the transfer learning or use external similar data source Kolesnikov et al. (2019). Instead search new model-oriented method which can be applied on new kind problems where no similar dataset exists.
HyperbanndDiijkstrra is an innovative way to benefit from this increasing computing power. It consists in unifying the two already proven efficient but contradictory approaches: hyperparameter optimiza- tion (HPO) and ensemble. First, explores and trains models until finding the optimal solution and wasting sub-optimal ones while the other one uses a population of trained models to predict more accurately.

---

Under review as a conference paper at ICLR 2021
SriperbandDijjsttee creates ensemble based on hyperband which is able to generate huge num- ber of trained deep models. Then, Dijkstra yields efficient combinations between them. As far as we know, it was never proposed to use Dijkstra's algorithm to find subset of le previously trained models in greater population.
After that, describe and discuss interesting properties and experimental results on two datasets:
Hyperband-Dijke is able to generate better ensemble than any ensemble of ResNet mod- els.
We show that Dijkstra algorithm is better to aggregate trained models than naive strategy consisting in taking the top k models based on their validation accuracy.
We show that our workflow (with ensemble of size 2 2) keeps benefiting of hyperband rumning after many days while standard use of hyperband (consisting in taking only the best model) stops improving much earlier.
2 RELATED WORKS
In this section we briefly review the main ideas from prior work that are relevant to our method.
Ensemble. Authors Sollich & Krogh 1995) laid the foundation stone about the idea that ever-fitted machine learning algorithms can be averaged out to get more accurate results. This phenomenon is explained by the Law of Large Numbers which claims that the average of the results obtained from large number of trials should be close to the expected value. These results are especially interesting for deep learning models because they are machine learning models which are the most affected to random effects (over-fitting) due to their huge amount of parameters.
Many ensemble algorithms have been invented such Wolpert (1992), Breiman (199) or boosting Schwenk & Bengio (2000). Some other methods are neural networks specific like negative correla- tion learning Liu & Yao (1999), dropout /Srivastava al. (2014) or snapshot learning Huang et al. 2017 There is today no consensus on the way to do ensembles like shown in the appendix
In case the architecture of models in the ensemble biased for example all models contained are not deep enough or not wide enough to capture relevant features in the data exploiting parametric diversity will not efficiently improve the results. That is why authors Liao & Moody (I999) Gashler et al. 2008) promote more and more diversity, not only based on the random weights initialisation but based on different machine learning algorithms such as neural network and decision tree in the same ensemble to maximize diversity and therefore the accuracy.
Knapsack problem. A Combinatorial Optimization problem consists in searching for solution in a discrete set so that a function is optimized. In many such problems, exhaustive search is not tractable, that is why approximate methods are used. Dijkstra's algorithm Dijkstra (1959) is a path finding algorithm which locally selects the next best node until it reaches the final node. A HHrrt et al. (1972) is an informed algorithm which first expands the most promising node to converge faster than Dijkstra. This knowledge is used only if an appropriate heuristic function is available. Otherwise, in absence of this knowledge, Dijkstra and A are equivalent. More recently, SP-MCTS Schadd et al. (2008) is probabilistic approach which runs many tree explorations based on the Upper Confident bound applied to Tree (UCT) Kocsis & Szepesvári (2006) formula to guide exploration//xxploittionn to catch a maximum of information on one node before selecting it.
Hyperparameter Optimization. The empirical nature of research in Deep Learning leads us to try many models, optimization settings and pre- processess settings find the best suited one for data. No Free Lunch theorem Wolpert & Macready (1997) proves that no hyperparameter optimization can show a superior performance in all cases. Nevertheless, methods have been developed and have shown stable performance supervised deep learning dataset.
Discrete-space search enables to search the best model description to a given neural network. Under this umbrella, we can find the number of units per layer, regularization parameters, batch size, type of initialization, optimizer strategy, learning rate. Plenty of approaches exist with a different cure-exploration approach |Bergstra & Bengio (2012). Li et al. (017)), smart computing resources allocation strategies Li et al. [2017) et al. (2018) a priori based Hoffman et al. (001), posteriori based Bergstra et al. (201)) or genetic inspired Jaderberg et al.
theoretical background,
2

---

Under review as a conference paper at ICLR 2021
2017). Those methods not exclusive, for example BOHB Falkner et al. (2018) mixes Bayesian Optimization strategy to Hyperband.
Another automatic approach exists like graph-space search Pham et al. (2018) It consists in finding the best architecture (graph) of neural networks. provides a maximum degree of freedom in the construction of the neural network architecture. Due the infinity of combinations, scientists implement several constraints to limit the possibilities graph generation, save computation cost and preserve the correctness of generated graphs. All hyper-parameters, like optimization settings and data pre-preprocessing are given by user to drive into this graph-space. Due to this complexity and because only models architectures are explored, we decide
follow this path.
Parallel hyperparameter optimization. All HPO strategies presented in this paper are asyn- chronous so their deployment is ideal on multi-GPU multi-node GPU HPC. Distributed client- server softwares Matthew Rocklin (2015) Moritz et al. (2018) allow to simultaneously spread those training candidate models and evaluate them. Those frameworks allow also serve them in parallel.
Multi-objective goal. Authors Johnston et al. (2017) discovered that many neural networks have comparable accuracy. Literature lets us imagine that the hyper-parameter function topology has two plateaus: where the optimizer algorithm converges and where it does not. This flatness can be used to optimize secondary goal such as model size, tome-to-pedicctinn power consumption and so on. Authors Patton et al. (2019) propose multi-objective optimization not only search an accurate model but also faster ones.
Early Stopping. A common practice exists to speed up HPO running like Early Stopping, They con- sists in resources reallocation strategies by considering learning dynamic DNN. Prechelt (1998) Li et al. (2017). Early stopping is also known to be regularization method that stops the training when the validation accuracy plateaus is symptomatic and that it will not generalize well (overfit- ting).
PROPOSED WORKFLOW
In this section we will first see the workflow proposed before going into more detailed explanation step by step.
3.1 DETAIL OF THE WORKFLOW
As shown figure L the proposed workflow consists in using hyperband and not only saving the best one on the disk but the sub-optimal one too. Second, a combinatorial optimization algorithm (Dijkstra's) finds the best one regarding the maximum number of models desired by the user (noted K). Dijkstra's 's algorithm computes the validation loss of candidates ensemble to evaluate how well a solution will generalize on the test database.
Train data
Validation data
Test
Max nb. models wanted
The best model
Sub optimal 1
The best ensemble
Sub optimal o
Sub optimaal
Hyperband
Sub opemal 2
Dijkstra Madel selection
Prediction
Prediction b
Averaging
Sub
prodiction
Figure 1: The algorithmic workflow to train, combine and predict on new data. this example K=2.
The final accuracy depends on the running time hyperband and the number of models chosen in the ensemble. Experiments results shown in section 4
3

---

Under review as a conference paper at ICLR 2021
The workflow introduce is simple. We use Hyperband algorithm and the distributed framework Ray Moritz et al. (2018) and then our combinatorial optimization Disjkstra's 's algorithm is a natural choice to ensemble models. The simplicity of the chosen algorithm and re-using existing frame- works reinforce our claims that this work is easy to test on a new dataset.
3.2 STEP HYPERBAND TO GENERATE MANY MODELS
Hyperband relies on an iterative selection of the most promising models allocate resources, al- lowing it to exponentially evaluate more configurations than strategies which do not use progressive results during training. Hyperband is technique that makes minimal assumptions unlike prior con- figuration evaluation approaches. Its pure-exploration nature combined with conservative resource allocation strategies can sweep better the hyperparameter space than other strategies like blackbox bayesian optimization. This diversity of models sampled are ideal to combine them for Dijkstra's algorithm and make better ensemble Liao & Moody 1999) Gashler et al. 2008).
We only store models trained at least half maximum epochs. This allows to reduce the number of models saved and thus the number of possible combinations by focusing on the most promising models explored by Hyperband.
3.3 STEP DIJKSTRA'S S ALGORITHM TO COMBINE MODELS
We discuss that finding the best combination of K among larger population is first modeled as graph. We then prove that no exact solution can be applied because of the computing complexity of the problem. That is why we propose Dijsktra's 's algorithm, simple and popular approach.
3.3. PROBLEM MODELING AND INTUITION
The solution space can be modeled as a tree with the empty ensembles as the root. Every node represents a deep learning model added and any path an ensemble. All nodes can be terminal and not only leaves. To evaluate and compare their score, we use the formula [] It calculates the cross entropy between validation labels, averaging predictions and the current ensemble I of size k.
scores = CF K
Σ 9i) TEI
Figure 2 is a simplified example of three models and their combinations on error distribution. The modeled tree associated is shown in figure The problem is to find the best combination be as close as possible to the center (0; 0). We observe that the best individual model (d(c) = 0.22) is not always the best one to combine with the others (d(a,)) 0.07). That is why smart algorithms
are needed. We also eliminate the simple idea that combining all models systematically leads to the best solution (d(a.be) 0.12).
3.3.2 PROBLEM COMPLEXITY
Finding the best ensemble of maximum size K among models with 1 S K < n. is case of the "knapsack problem' belonging to the NP (non-deterministic polynomial) problem's family. There is no known exact method except brut-forcing all possibilities.
The number of possible subsets of size k among n items is computed with the binomial coefficient. This binomial formula is known Das 2020) to be asymptotically polynomial when n is increased and & is fixed. When the user puts the maximum size ensembles K to explore, all combinations l such 1 < < K are also candidates. Therefore the quantity of candidates ensembles is given by ΣK (2) This formula also has polynomial behavior when k is fixed and n increases. For example, for K = 4 and a population of n 100, adding only one new model in the population increases the number of combinations from 4.09 million to 4.25 million.
This polynomial behavior has two exceptions: when K = 1 (linear) and when K = n (exponential). K = n allows the research of big ensembles for a maximum precision despite inference time. The number of ways to construct a non-empty model from catalogue of N models is formally described by the equation Σd= (N) 2N 1. We have two options for each model using it not (
4

---

Under review as a conference paper at ICLR 2021
(1)
(1)
The two represent error of Figure Available solutions as a graph. Nodes two different classes, the goal is to get closer ensembles. Edges the decision to add new model to the ensemble. The cost function is the cu- different error distribution and averaging them leads to ensembles with other error distribu- each node. The source node, corresponding to the tions.
Figure 2:
to the (0:0) point. Each model a, b and have clidean distance and is displayed the top left of
empty ensemble, modeled with arbitrary large distance. The optimal ensemble is made of a and b.
possibilities). We exclude the empty ensemble (-1). It means that for each new model found by hyperband, the quantity of combinations is multiplied by 2. The combination of ensembles with catalogue of 100 models is = 1.
Due to this combinatorial explosion, we understand the need of an approximation search algorithm.
3.3.3 APPROXIMATE SOLUTION WITH DUKSTRA'S ALGORITHM
This huge number of ensembles combined to the fact that relationships between model predictions and labels are complex (figure 2 and formula [ and that no heuristic is known makes Dijkstra's 's al- gorithm a natural choice for this class of problems Voloch (2017). Dijkstra's 's algorithm Dynamic Programming procedure, meaning it makes and memorizes successive approximation choices.
While the number of possibilities requires approximate solutions, this huge number of candidate ensembles has the advantage ensuring that better combinations should be found compared to a naive aggregation approach. This is confirmed in the results in section
Once model is found based on the running by Disjkstra's 's algorithm, we can combine predictions of models on new data evaluate it on the test dataset. As training and evaluating, models predicting can be distributed on different GPUs but the averaging require all models finished their prediction.
4 EXPERIMENTS AND RESULTS
We experiment our workflow on CIFARIOG 100 Krizhevsky (2009) and microfossils datasets both pre- sented in appendix [.33 On these datasets there are 16 hyper-parameters explore. Experimen- tal settings are explained in appendices B for reproducibility purpose but this level of detail is not required understand our works or results. possible that larger hyperparameters value ranges may positively influence again the results obtained.
In this section, we evaluate different workflows by evaluating various HPO strategies, different combinatorial optimizations. We also different settings like the number of models in produced ensemble and effect of HPO running time on results.
4.1 VISUALIZATION OF ARCHITECTURE SAMPLING RESULTS
As others have highlighted, no correlation exists between the accuracy and computing cost of model on image recognition. We display it in the figure results random sampling of hyperpa- rameter space the CIFAR100 dataset. This is the reason why we propose target function which measure efficiency of one model based on its accuracy and its inference time. The implemented formula is: CF ū) W/ with / the inference time 2,000 images expressed in seconds and W
5

---

Under review as a conference paper at ICLR 2021
scaling factor arbitrary choosen such as W 0.001. Hyperband minimizing this target function increases the concentration of computing resources on more efficient models. The natural efficiency of Hyperband combined to the fact that we use this multi-criteria target function allows to increase the number of models explored in 6 days by factor 3.2 compared to Random Sampling + Early Stopping (plateau detection).
Another Early Stopping method used consist in detecting after one epoch the models perform better than random predictions on the validation dataset. It shows very effective detect early which models are unable to learn and free GPUs for other models. Experiments in figure 5 show that about 14% of models diverge so we can save quasi-entirely their running.
Inference time (sec.)
Frequency
0:325
0.00
0.05
10%
Accuracy
Figure 4: Correlation computing cost versus Figure 5: Accuracy histogram of randomly sam- accuracy of randomly sampled models on CI- pled models on CIFAR100 FAR100
Accuracy
4.2 COMPARISON OF VARIOUS HPO STRATEGIES AND DUKSTRA'S ALGORITIIM
We evaluate the accuracy of our workflow CIFAR100 by replacing Hyperband with various HPO strategies in [ Retraining the same deep learning architecture from scratch can yield significant distance in differrnn run time, that why we compare different HPO strategies and different popular ResNet architectures as well in table 2 We observe that Hyperband generally performs well to take the best one and also to aggregate ensembles compared all other methods. confirms our claim in the previous section on Hyperband computing efficiency and the ability to generate good ensembles.
We also observe that most of HPO strategies discovered better models than the best ResNet models found. For both benchmarks, we observe that ResNet18 compared to other ResNet architectures, lead to better models but when we combine them, ResNet34 is a better choice. We conjecture that ResNet18 leads to lower parametric diversity compared to ResNet34 models because of the lower number of layers.
Another remark is that proportion of 14% of randomly chosen models diverge while 100% of ResNet models converge. It shows that ResNet are robust handcrafted models but a random process find more accurate
new dataset.
The same results and conclusion on the microfossils dataset are reached in tables B and
Table 1:
Various HPO strategies and various ensemble size computed on the CIFAR100 dataset
RS
HB
BO
TPB
BOHB
PBT
Top
68.74%
70.69%
68.22%
69.83%
70.56%
60.82%
Team of
72.36%
74.55%
71.60%
73.79%
74.20%
65.27%
Team of
74.73%
76.43%
73.09%
74.36%
76.07%
67.90%
Team 4
77.01%
77.11%
73.60%
75.55%
77.22%
68.87%
Dijkstra solution
Team of
78%
78.45%
75.02%
76.77%
77.94%
69. 80.80%
Team
78.89%
78.89%
77.33%
78.16%
70.73%
Team of 12
79.30% %
79.18%
76.04%
77.25%
78.60%
71.09%
79.36%
79.44%
76.04%
77.34%
78.70%
71.39%

---

Under review as a conference paper at ICLR 2021
Table 2: Comparison different ResNet populations and ensemble size the CIFARTO0 dataset
resact population
resnct34 population
resact50 popuiation
resncXt50 population
resnet101 population
resnct152
population
Top
65.70%
63.23%
63.37%
58.72%
65.15%
63.18%
of
02%
69.30%
40%
62.33%
66.14%
Team of
70.12%
71.42%
69.71%
64.84%
69.70%
67.84%
Dijkstra solation
Team of 71.34%
72.37%
70.24%
65.91%
70.26%
69.35%
Team o
72.30%
73.12%
71.84%
67.85%
71.07%
69.74%
Team of 8
73.02%
73.79%
72.77%
68. 14%.
71.74%.
70.675
74.22%
70.54%
Team of 16
73.44%
74.26%
73.20%
68.06%
71.85%
70.55%
Table 3: Various HPO strategies and ensemble size on the microfossils dataset
Top
86.55%
88.07%
85.56%
86.09%
87.82%
85.10%
Team of
89.02%
90.18%
89.07%
89.34%
88.29%
86.73%
89.16%
90.07%
Dijkstra solution
Team of 4 89.77%
91.09%
90.07%
90.94%
90.166
88.01%
Team
90.30%
91.13%
90.79%
90.70%
90.72%
Team of 8
90.61%
91.54%
91.23%
90.79%
91.20%
90.12%
4.3 HYPERBAND AND VARIOUS COMBINATORIAL STRATEGIES
Different combinations algorithms tested in figures 6 and by varying the number of models from to 16. The total population models only contains models trained during at least 50 epochs by Hyperband.
We tested two naive strategies. The first one consists in drawing randomly ensembles of K models and the second one in taking the top-K. Dijkstra's algorithm generally finds better solutions than naive strategies.
We also evaluate SP-MCTS, the solution space was modeled as an unfolded tree representation leading to nodes redundancy, so equivalent nodes were implemented to index the same score. Based on preliminary experiments, SP-MCTS is set to run 1000 x K with K the maximum desired number of models to favor accuracy over SP-MCTS computing cost. With single-threading implementation, Dijkstra's 's algorithm takes only 25 seconds to find an ensemble of K = 16 among 66 models while SP-MCTS is x580 slower.
search algorithm based on Monte-Carlo. To test SP-MCTS,
In the microfossils dataset, Dijkstra's algorithm falls to local minimum and uses the same 10 models when K > 10. SP-MCTS do not falls into this trap and keeps benefiting of an increasing power.
60.95
60 40.00%
8036
71.3%
50.995
75.96
74.0%L
72.00%
12.39%
Top indiddial modes
so: vm..
1o3 inhedse modds
10.3%
62.305
landon quartlo
07.09
sandon tuartlos
sandon quartiee
64.0%
62.005
2 5 7 15 15
Nb. of the ancemble
Figure 6: The CIFAR100 dataset
Figure The microfossils dataset
Figure 8: Different combinatorial optimization algorithm tested

---

Under review as a conference paper at ICLR 2021
Table 4: Various ResNets and ensemble size the microfossils dataset
resnet 18
reanet34 population
resnet50 population
reaneX130 population
resnet101 population
reanet152
population
population
Top
86.83%
85.24%
85.65%
84.91%
84.22%
85 85.09%
Team of
87, 97.74%
87.77% %
87.54%
87.26%
86.04%
86. %
Dijkstra solution
Team
88..2%%
88.55
88.4t
87.93%
87.54%
66.6%%
Team of 4
88, 34%
88,71 %
88.63%
88.02%
87.74%
87.91%
Team of
89.10%
89.50 50
89,37%
88.37%
88.24 29.244%
88.78%
Team of 8
89.13%
89.59 %%
89.39%
88.73%
88.40%
88.8018
4.4 EFFECT OF COMPUTING INTENSITY ON THE FINAL ACCURACY
Our workflow benefits more computing intensity than standard Hyperband like shown in figures 9 and 10
After 24 hours, standard Hyperband (consisting in taking only the best model) converges while our worflow with K keeps benefiting of the models generated. On the CIFAR100 dataset, identify that ensembles of 12 and 16 models benefit linearly of the computing time. Their accuracy begins 77% and increases of +0.4% every 24h00.
Moreove, we observe that adding more models systematically leads to an increasing of accuracy but this trend declines. We show that the benefit obvious from to 2 models (+3.9%) but the improvement is small from 6 to 16 (+1%).
125
135
70h
154
305
Him myttc
125
right
1154
Four moduss
Three model
Three moddey
Two model
606
Figure 9: The CIFARI00 dataset
Figure 10: The microfossils dataset
Figure 11: Varying the max number of models in ensembles in function of Hyperband running time
CONCLUSION
Due to the experimental nature of deep learning and the increasing of available computing power like multi-Gii GPUU servers, allows sweep deep learning architectures. The standard usage of hyper parameter optimization consists in training hundreds of models and keeping only the best one, which leads to an important waste of energy.
We show that Hyperband efficiently generates diverse architectures coupled by significant number of combinations between them. That is the reason why a smart selection of models with Dijkstra allows to build accurate ensembles. This workflow benefits of the increasing computational power and proposes a general approach to unify datasets, it has been also showed that our ensembles more accurate than naively build ensemble. Our workflow also yields an ensemble more accurate than any other ensemble of ResNet models.
search and resembling models. On two
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthicu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg. Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learn- ing on heterogeneous systems, 2015. URL https: /tt:::/wwww.. tensorrerrooooo oo org/ Software available from tensorflow.org.
James Bergstra and Yoshua Bengio. Random search for hyper-parameter optimization. J. Mach. Learn. Res., 13:281-305, 2012. URL http://dblp. wnw.uniiitteeeeeee.. wtt....eeddd///uunneee.ee
m jmmrllll33 ht htt.shhttattaaa1111
James S. Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. In Shawe-Taylor, R. S. Zemel, P. L. Bartleu, F. Pereira, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 24, pp. 2546 2554. Curran Associates, Inc., 2011. URL http:://apees:: nips happp...o cc/paper/ 4443- a moorithms--o for for-hyer -aa-ppaaaae parameter opt zation. pdf
Leo Breiman. Bagging predictors. Machine Learning. 24(2):23-140,, Aug 1996. ISSN
1573-0565.
doi: 10.1023/:10188053344335 URL https://doi. 1023/A/
101 8054314350
Shagnik Das. A brief note on estimates of binomial coefficients.
http://page .miagemmi.
Fu-berl in de/ whaagii//otes//bnnnntt pdd. 2020. september-2020j.
[Online; accessed 25-
E. W. Dijkstra. A note problems in connexion with graphs. Numer: Math., 1(1):269-77. December 1959. ISSN 0029-599X. doi: 10.1007/BFF1188690. 10. N0.1007/BF0138633
URL
Stefan Falkner, Aaron Klein, and Frank Hutter. BOHB: robust and efficient hyperparameter op- timization at scale. CoRR, abs/1807.01777 2018 URL
01774
Michael Gashler, Christophe Giraud-Carrier, and Tony Martinez. Decision tree ensemble: Small heterogeneous is better than large homogeneous. 2008 Seventh International Conference Ma- chine Learning and Applications, pp. 900-905, 01 2008. doi: 10.1109/ICMLAA2000 154.
Felipe O. Giuste and Juan C. Vizcarra. Cifar-10 image classification using feature ensembles, 2020.
Antonio Gulli and Sujit Pal. Deep learning with Keras. Packt Publishing Ltd, 2017.
Peter E. Hart, Nils J. Nilsson, and Bertram Raphael. Correction to "a formal basis for the heuristic determination of minimum cost paths". SIGART Bull., (37):28-29 December 1972. ISSN 0163- 5719. doi: 10.1 00.144//055777.1066777 1056779
URL / 1tt.:o/////1111111151555111
do
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CoRR,
03385
Matthew Hoffman, Eric Brochu, and Nando de Freitas. Portfolio allocation for bayesian optimiza- tion. In Proceedings of the Twenth-Seventh Conference on Uncertainty in Artificial Intelligence, 1AI' pp. 327-336, Arlington, Virginia, USA, 2011. AUAI Press. ISBN 9780974903972.
Gao Huang, Yixuan Li. Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Weinberger. Snapshot ensembles: Train 1, get for free. CoRR, abs/1 htss/704.000009 2017. URL http:// arxiv. Doxivoorg/abs/1100 00100

---

Under review as a conference paper at ICLR 2021
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning. Karen Simonyan, Chrisamtha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. CoRR, abs/1711.09846, 2017.
/1711. 09846
Travis Johnston, Steven Young, David Hughes, Robert Patton, and Devin White. Optimizing convo- lutional neural networks for cloud detection. pp. 1-9, 11 2017. doi: 10. 10.114553144447:3344
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 12 2014.
Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. volume 2006, pp. 282- 293, 09 2006. doi: 10. 1007/1187884229)
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit-based approach hyperparameter optimization. 18(1):6765-681 January 2017. ISSN 1532-4435.
Yuansong Liao and John Moody. Constructing heterogeneous committees using input feature group- ing: Application to economic forecasting. pp. 921-927, 1999.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Sto- ica. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:1807.05111: 2018
Y. Liu and X. Yao. works, 12(10):1399 1404, 1999. ISSN 0893-6080.
Ensemble learning via negative correlation. Neural Nei- doi: https://do..rr//10 1016/00889--6000099)00777...., URL http: /tt::/wwww www www.sscieneeirrett.... com/science/
articcle/p11/0099560009999
Matthew Rocklin. Dask: Parallel Computation with Blocked algorithms and Task Scheduling, pp. 126 132, 2015. doi: 10.250800Maajor2-7998:: 7h9erred--7
Philipp Moritz, Robert Nishihara, Stephanie Wang. Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael Jordan, and Ion Stoica. Ray: A distributed framework for emerging ai applications. In Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation, OSDI' 18, pp. 561-577, USA, 2018. USENIX Association. ISBN 9781931971478.
Robert M. Patton, J. Travis Johnston, Steven R. Young, Catherine D. Schuman, Thomas E. Potok, Derek C. Rose, Seung-Hwan Lim, Junghoon Chae, Le Hou, Shahira Abousamra, Dimitris Sama- ras, and Joel Saltz, Exascale Deep Learning to Accelerate Cancer Research. arXiv e-prints, art. arXiv:1909.12 122.99 Sep 2019.
Hieu Pham, Melody Guan, Barret Zoph, Quoe Le, and Jeff Dean. Efficient neural architecture search via parameter sharing. CoRR, abs/1802.03268, 2018. URL mtlhattp///axxii org/ abs/
18C2.03263 03268
Lutz Prechelt. Early stopping- when? In Neural Networks: Tricks of the Trade, This Book is an Outgrowth of a 1996 NIPS Workshop, pp. 55-69, Berlin, Heidelberg, 1998. Springer- Verlag.
ISBN 3540653112.
Maarten Schadd, Mark Winands, H. Herik, Guillaume Chaslot, and Jos Uiterwijk. Single-player monte-carlo tree search. pp. 1-12, 09 2008. doi: 10.10779988-35540-87666 1.
Holger Schwenk and Y. Bengio. Boosting neural networks. Neural computation, 12:1869-8 87,, 09 2000. doi: 10. 10.1 00.112208997770000011111111
10

---

Under review as a conference paper at ICLR 2021
Peter Sollich and Anders Krogh. Learning with ensembles: How overfitting can be useful. 8: 190-196, 01 1995.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1): 1929-1958, January 2014. ISSN 1532-4435.
Nadav Voloch. Optimal paths of knapsack-set vertices on weight-independent graph. WSEAS Transactions on Computers, 16:163-171, 09 2017.
D. H. Wolpert and W. G. Macready, No free lunch theorems for optimization. Trans. Evol. Comp, 1(1):67-88 82, April 1997. ISSN 1089-778X. doi: 10.1 10.1109/42355588893. URL https://doi.
ozc/10..11094225..555 585893
David H. Wolpert. Stacked generalization. Neural Networks, 5:241-259, 1992.
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated resid- transformations for deep neural networks. CoRR, abs/1611.05431, 2016. URL http:
// wwapxii..rr/abb/11111. /at../abba1111....
Sergey Zagonukko and Nikos Komodakis. Wide residual networks. CoRR, abs/1605.07146, 2016. URL htA.hppp//arri...or///////00.000

---

Under review as a conference paper at ICLR 2021
A
ENSEMBLE AWARDS IN PAST PUBLIC IMAGE RECOGNITION CHALLENGES
URL http://dde.mmekkagggaapa-----loi
Name: Understanding Clouds from Satellite Images
Description: Can you classify cloud structures from satellites?
Participation: 1538 teams
Prize: $10K
Dead line: 2019-11-19
1st place Use averaging predictions of segmentations models; averaging segmenta-
tions models
3rd place Use majority vote of segmentations models
4th place Averaging of 10 models
5th place Weighted averaging of segmentations models
Name: RSNA Intracranial Hemorrhage Detection
Description: Identify acute intracranial hemorrhage and its subtypes
Participation: 1345 teams
Prize: $25K
Dead line: 2019-10-28
2st place 15 bagging LSTMs (3 bootstraps)
3rd place Use weighted averaging predictions of 17 models
5th place Stacking of 9 segmentation models
Name: Lyft 3D Object Detection for Autonomous Vehicles
Description: Can you advance the state of the art in 3D object detection?
Participation: 547 teams
Prize: $25K
Dead line: 2019-11-3
3rd place - 3 faster-renn models with Soft NMS
Name: Severstal Steel Defect Detection
Description: Can you detect and classify defects in steel?
Participation: 2431
teams
Prize: $120K
Dead line: 2019-10-18
1st place Ensemble of 4 classifications (which one ?): Ensemble of 9 segmenta-
tions(which one ?)
4th place Ensemble of segmentations (which one ?)
Name: Kuzushiji Recognition
Description: Opening the door to thousand years of Japanese culture Participation: 293 teams
Prize: $15K
Dead line: 2019-10-15
1st place Ensemble of 2 R-CNN
2nd place 1 RCNN, Stacking with an ensemble of XGBoost and LightGBM aver-
aging
3rd place Hard voting of models; NMS with 2 models
Name: The 3rd YouTube-8M Video Understanding Challenge
Description: Temporal localization of topics within video
Participation: 283 teams
Prize: S25K
Dead line: 2019-10-04
1st place 17 averaging models smooth out predictions
2nd place models weighted averaging, weights fixed manually
3rd place Stacking 12 models
Name: APTOS 2019 Blindness Detection
Description: Detect diabetic retinopathy to stop blindness before it's too late
Participation: 2931 teams
Prize: S50K
1st place Ensemble of 8 models with stacking
4th place Averaging of models
12

---

Under review as a conference paper at ICLR 2021
B REPRODUCIBILITY
B.1 HARDWARE
Computing nodes used are the same than the Oak Ridge Summit nodes. 6 Tesla-V100 GPUs was used for running each hyperparameter optimization algorithm and generate required populations.
B.2 SOFTWARE STACK
Hyperparameter optimization framework Tune Liaw et al. (2018) was used. It runs above the Ray client-server Moritz al. (2018) witch spread experiments ing and data augmentation was coded with the framework Keras Gulli & Pal (2017) with backend Tensorflow 1.14.0 Abadi al. (2015).
GPUs. Deep Learning train-
B.3 THE DATASET USED
The CIFAR100 dataset. C1FAR100 Giuste & Vizcarra (0200) consists to 60,000 32x32 RGB images in 100 classes. For each class, there are 580 training images, 20 validating images and 100 testing images.
The Microfossils dataset. Microfossils are extremely useful in age dating, correlation and paleo- environmental reconstruction to refine our knowledge of geology. Micro-fossil species are identified and counted on large microscope images and thanks to their frequencies we can compute the date of sedimentary rocks.
To do reliable statistics, a big amount of objects needs to be identified. That is why we need deep learning to automate this work. Today, between 400 and 800 fields of view (microscopy imagery) need to be shot for 1 rock sample. In each field of view, there are between 300 to 400 objects to identify. Among these objects, there are non-fossils (crystals, rock grains etc...) and others fossils that we are looking for to study rocks.
Our dataset contains 91 classes of 224x224 RGB images (after homemade preprocessing). Micro- fossils are calcareous objects took with polarized light microscopy. The classes are unbalanced. We have from 50 images 2500 images by class, with total of 32K images in all the dataset. The Fl score was used and labeled as 'accuracy' on all benchmarks.
B.4 HYPERPARAMETES CONFIGURATION SPACE
The table shows all hyperparameters properties in this workflow. We use a ResNet Zagoruyko & Komodakis 2016 based architectures due to its simplicity to yield promising and robust models on many datasets. We explore different residual block versions: "V1". "V2" He et al. (2015) and "next" adam optimizer Kingma & Ba (2014) due to its well known performance and its low learning rate tuning requirement. Hyperparameters be updated during the training, for example the learning can change
Xie et al. (2016) Regarding the optimization method,
labeled as "mutable" but the architecture cannot. PBT algorithm is the only one algorithm tested to discover schedule of mutable hyperparameters.
We aware that our research may have a limitations. The range of hyper-parameter can be to short compared to good results found in the literature Zagoruyko & Komodakis (2016) like the batch size, width and depth of convolutionnal neural network. Moreover we could explore other optimization strategies like SGD with momentum and also the learning rate decay. Next, dropout also a promis- ing method we could explore. To finish, on CIFAR 100 our maximum number of epochs is 100 and scientists before us usually use 160 epochs.
B.5 ADAPTATION TO APPLY ON CIFARI00
The CIFAR100 dataset contains 32x32 images while usually ResNet adapted to be used on imagenet (224x244 images). Those different resolution need some adaptation. On the CIFAR 100, the first convolutionnal network replaced from the 7x7 kernel size with a stride of 2, 3x3 kernel size with a stride of
13

---

Under review as a conference paper at ICLR 2021
Table 5: The hyperparameter space experimented
Name
Type
Range
Mutable
Learning rate
Continuous
[0.001, 0.01] (log space)
Optimization
Batch size
Discrete
[8; 48]
yes
L2 regularization factor
Continuous
[0:0.1] (log space)
Convolution type
Categorical
("v1", "v2", "next")
Activation func- tion
Categorical
("tanh", "relu", "clu")
Neural network architecture
No
Number of filters in the first convolu- tion layer
Discrete
32, 64, 96, 128
Number of filters in other convolu- tion layers
Discrete
32, 64, 96, 128
Number of convo- lutionnal block in the first stage
Discrete
1, 3, 4, 6, 8,
Number of convo- lutionnal block in the second stage
Discrete
1,2, 4, 6, 8, 11
Number of convo- lutionnal block in the third stage
Discrete
1, 3, 4, 6, 8, 11
Number of convo- lutionnal block in the fourth stage
Diserete
1,2, 4, 6, 8, 11
Max zoom
Continuous
[O; 0.6]
Data augmentation
Max translation
Continuous
[O; 0.6]
yes
Max shearing
Continuous
[O; 0.3]
Max channel shift-
Continuous
[O; 0.3]
ing
Max rotation mea- sured in degrees
Discrete
[0; 90] (log space)
14


Reviews:
[{"review_id": "zC2P2AXUoqV", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposes the use of Dijkstra's algorithm for ensemble construction, where the pool of classifiers to chose from is models trained during a hyperparameter optimization procedure. Authors show that the performance of the constructed ensembles improves upon the performance of a single classifier. \n\nStrong points:\n\n- Novel and interesting idea\n\nWeak points:\n\n- The paper would benefit significantly from proofreading and copyediting.  \n- The choice of the Dijkstra algorithm is poorly justified and might be incorrect\n- More baselines for comparison are required\n\nRecommendation:\n\nI strongly recommend that this paper be rejected. While the idea of using Dijkstra for post hoc ensemble construction is interesting, I don't think it is justified and I am not convinced that it is a correct choice. Furthermore, the paper is riddled with mistakes, unneeded details and bad explanations. This paper needs to be entirely rewritten and the experiments require proper baselines for post hoc ensemble construction. \n\nExtra comments:\n\nThere is no justification provided for using Dijkstra. The context is different from shortest path searching, as multiple classifiers in an ensemble have complex interactions that are lost via the majority voting and empirical risk estimation. In other words, the \"weight\" of the edge from one node to another will vary depending on the nodes visited beforehand, which is not a condition normally present for shortest path estimation. I think it means that it's incorrect to apply Dijkstra here, or at least that Dijkstra offers no guarantees. The same goes for the parallel with the Knapsack problem (i.e. I don't think this is an instance of the Knapsack problem).\n\nOther baselines should be added, some simple ones would be forward/backward greedy search, stacking and genetic algorithms. \n\nSection 3.3.1: showing with a single example that combining three models with a majority vote does not lead to optimal performance is hardly evidence that \"smart algorithms are needed\".\n\nThe related works section is needlessly long. For example, no need to mention work on multi-objective optimization. \n\nFormatting of citations is incorrect, see the conference author guidelines.\n\n\nMissing citations:\n\nThe correct citation to introduce boosting is \"Y. Freund, and R. Schapire, “A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting”, 1997.\" not Schwenk & Bengio. \n\nAmongst others, Caruana et al. (2004), Feurer et al. (2015) and Levesque et al. (2016) have already applied ensembling to hyperparameter optimization, such references are missing from your relevant works section. \n\nReferences: \n\nCaruana, R., Niculescu-Mizil, A., Crew, G., & Ksikes, A. (2004, July). Ensemble selection from libraries of models. In Proceedings of the twenty-first international conference on Machine learning (p. 18).\n\nFeurer, M., Klein, A., Eggensperger, K., Springenberg, J., Blum, M., & Hutter, F. (2015). Efficient and robust automated machine learning. In Advances in neural information processing systems (pp. 2962-2970).\n\nLévesque, J. C., Gagné, C., & Sabourin, R. (2016). Bayesian hyperparameter optimization for ensemble learning. arXiv preprint arXiv:1605.06394.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XD_9hmU84Zr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After rebuttal: no rebuttal, so I will keep my score.\n\n-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nOverview: This paper proposes to apply ensemble methods to make use of sub-optimal configurations saved from Hyperband. In particular, the authors propose to use Dijkstra's algorithm to choose the best combination of size K among n models.\n\nReasons for the score:\nOverall, I vote for reject. There are several reasons that will be detailed a bit in the next. Roughly, I think the paper requires a huge effort of rewriting before being published. Beyond that, I'm not fully convinced by the technical novelty or difficulty of this paper.\n\nPros:\n- The idea of collecting sub-optimal configuration to boost the performance has not been considered in the context of HPO (to the best of my knowledge).\n- The experiments in Table 1 and Table show that the workflow do improve the final performance.\n\nCons:\n- My  first major concern is on the writing:\n  - There is no clear problem formulation of HPO.\n  - Personally, I'm not very comfortable with the authors not presenting formally what is Hyperband, what is Dijkstra's algorithm, and also how is the problem of finding the best ensemble modeled as a knapsack problem. I think these details can at least be provided in the appendices. I don't think it's appropriate to assume that readers are all familiar with every notion presented in the paper. Especially the main purpose of this work is to propose a new workflow that combines existing techniques, providing enough detail on how each component works and how they are connected in a precise way seems essential to me.\n- Now regarding the experiments:\n  - First of all, a general question, how many trials have you run for each experiment?\n  - Also, I don't see error bars on figure 6, 7, 9, 10.\n  - Could you explain a bit more why are you interested in the experiments of Table 2? I don't understand very well why it is useful to support the purpose of this work.\n  - Could you please more precise on when do you stop the experiments? It looks like for some experiments, a fixed time horizon is given. I guess it's the same for others, but again, some precise and formal descriptions are more than welcomed.\n\nGeneral questions and remarks:\n- Do you have any intuition on why you use averaging other than other ensemble strategies?\n- Although ensemble has not been considered in the context of HPO, I'm not sure that the scientific contributions in this paper is significant enough. Could the authors highlight a bit more the technical difficulties if I have missed anything?\n- In general, I feel like the paper is more like an engineering trick than a scientific discovery, which could be a nice contribution of course. But then in that case, I think more solid experiments should be provided. For example, in the context of HPO, we often want to see the evolution of performance over time, not only the final performance.\n\nMinor comments and grammar issues (non-exhaustive):\n- In general, I would suggest the authors to review a bit the writing style of the paper. Sometimes I feel like the authors have personal claims regarding some previous work without any support (which can be citations or even some intuitions). To cite one of them as example, in Section 3.2: Its pure-exploration nature combined with conservative resource allocation strategies can sweep better the hyperparameter space than other strategies like blackbox bayesian optimization. Maybe, but how is it compared to evolutionary algorithms for example? My point is that we should be careful about this kind of claims.\n- The citation style is weird, authors can refer to Section 4.1 of the template file of ICLR.\n- In the abstract: capable of doing sth. instead of capable to do sth.\n- Section 2, paragraph Multi-objective goal, I don't really understand the sentence: Literature lets us imagine that the hyper-parameter function topology has two plateaus.\n- Section 3.2: bayesian -> Bayesian.\n- Section 3.3.3: hyperband -> Hyperband.\n- Section 4.1: It shows very effective to detect... <- this is not correct grammatically.\n- Section 4.3: Different combinations algorithms -> Different combinations of algorithms.\n- Section 4.3: SP-MCTS do not falls into -> SP-MCTS does not fall into.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6suwpJcQd3g", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors contribute the idea of selecting a subset of models for building ensembles using the Dijkstra algorithm. Furthermore they apply this algorithm on the models produced by Hyperband (HB) and also show that it performs well on other hyperparameter tuning methods, such as random search and Bayesian optimization.\n\nGiven that the latter is just an application of the algorithm, the former will have to be the main contribution of the paper.\n\nWhat is unfortunately lacking is a comparison with other ensemble techniques including simpler ones. I am not an expert, but I know that simple greedy methods perform quite well, see for example [1]. So it it not clear the algorithm is any better than previous techniques. One advantage is that the proposed technique naturally incorporates a inference time constraint, but then it should be discussed.\n\nWithout a discussion and comparison to other widely using ensemble techniques this paper is incomplete and should not be accepted. The application alone on HPO technique is a fairly straightforward one, even if effective, and does alone not warrant a paper, with the decent but limited results presented. A stronger case could be made if for example the results allowed to reach top rank in many kaggle competitions, or in some existing AutoML competition.\n\n#### Pros\n* The paper is well written, polished, and easy to understand\n* Combining hyperparameter tuning and ensembling is a good idea. Although already routinely done, for example in Kaggle competitions and in AutoML competitions, it deserves more attention and analysis.\n* The results although limiting (only 2 datasets are used, ensemble baselines are missing) are promising.\n* Incorporating the cost constraint is an important contribution, and should perhaps be discussed in more detail.\n\n\n#### Cons\n* There exist many ensemble selection techniques in the literature, that are simple and known to work well, for example greedily selecting the next method that performs best if combined with the currently selected ensembles. In some sense the proposed algorithm can be considered the obvious extension of this idea if presented with a cost constraint. But it would be good to better discuss how a cost constraint changes the situation.\n* Too much focus on Hyperband, presented as part of the main contribution, I think the paper should focus more on their ensemble technique as it can be combined effectively with any method, as the results show, and just not that Hyperband seems to work best on the two datasets selected. Part of this seems to stem from Hyperband also working best when no ensembling is used, which is not necessarily in agreement with other HPO literature.\n* The paper would be stronger if results were presented on more datasets.\n\n[1] G Tsoumakas, I Partalas, I Vlahavas - A taxonomy and short review of ensemble selection\n\n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2KsNtSfb9wR", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is proposing to build ensembles of deep models, components of which have different hyperparameter (HP) configurations. This is done by first running Hyperband to create a large pool, and then run a greedy algorithm to construct an ensemble. This algorithm is termed Dykstra's algorithm on a certain graph, but it is of course simply just the default greedy algorithm, which is almost by default used to create an ensemble from a pool. The correct reference for this is [1], and this is just what people do when they create ensembles. The paper also misses a number of relevant recent work to build ensembles of deep models, at least [2, 3]. There is nothing new here, except maybe that Caruana's algorithm can now also be called Dykstra's.\nIn the very unlikely case I missed something, and what they call Dykstra's method here (not detailed in the paper) is different from the obvious greedy method of Caruana, then the paper fails by not comparing against this obvious baseline.\n\n[1] Caruana, Niculescu-Mizil, Crew, Ksikes\n  Ensemble selection from libraries of models\n  ICML '04: Proceedings of the twenty-first international conference on Machine learning\n[2] Deep ensembles:\n  B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (NIPS), pages 6402–6413, 2017.\n[3] Batch ensembles:\n  Y. Wen, D. Tran, and J. Ba. Batch ensemble: an alternative approach to efficient ensemble and lifelong learning. arXiv preprint arXiv:2002.06715, 2020\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Following a strong consensus across the reviewers, the paper is recommended for rejection.
They have all acknowledged some weaknesses of the paper, for instance

* Inadequate reference to prior work
* Unsatisfactory level of polishing
* Too limited evaluation, with more comparisons to baselines required
* The proposed approach ("Dijkstra algorithm") is not enough justified and motivated 
* Clarity (missing definitions of key components).

This list, together with the detailed comments of the reviewers, highlight opportunities to improve the manuscript for a future resubmission.

Decision: Reject