Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper discusses the use of post hoc ensembling in AutoML systems and proposes the use of a gradient-free numerical optimization method called Covariance Matrix Adaptation Evolution Strategy (CMA-ES), comparing its performance with the commonly used Greedy Ensemble Selection (GES) method. The authors apply these methods to AutoGluon on 71 classification datasets from the AutoML Benchmark and show that the choice of metric (balanced accuracy or ROC-AUC) affects whether GES or CMA-ES performs better. Specifically, for the metric balanced accuracy, CMA-ES outperforms GES, while for ROC AUC, GES outperforms CMA-ES due to overfitting. Additionally, the authors propose a method to avoid overfitting for CMA-ES, which involves a novel normalization method inspired by GES's implicit constraints during optimization. This method allows CMA-ES to perform as well as GES while keeping the ensemble size small.
overall_review: Strengths:

-The paper presents a detailed analysis of the performance of two post hoc ensembling methods, GES and CMA-ES, and compares their performance on 71 classification datasets from the AutoML Benchmark. This comparison provides valuable empirical contribution to the field of AutoML.
-The authors propose a novel way to avoid overfitting in post hoc ensembling using CMA-ES, which involves a normalization method inspired by GES's implicit constraints during optimization. This method can improve the performance of CMA-ES on the ROC AUC metric, which is known to be prone to overfitting.
-The paper provides insights into the effect of different validation data qualities on post hoc ensembling performance, highlighting the importance of using high-quality validation data.

Weakness:

-The paper does not apply the methods to other AutoML systems and limits the analysis to AutoGluon. Extending the evaluation to other AutoML systems would provide more generalizable insights.



review_summary: The paper provides a detailed comparison of two post hoc ensembling methods, GES and CMA-ES, and proposes a novel approach to avoid overfitting in post hoc ensembling using CMA-ES. Additionally, the paper provides empirical evidence to highlight the importance of using high-quality validation data for better results in post hoc ensembling. While there are some limitations to the paper, the insights provided are valuable for practitioners in the field.
actions_required_to_increase_overall_recommendation: NA
review_title: null

final_score: rating: 4: Weak Reject: For instance, a paper with minor technical flaws, limited impact, and/or weak evaluation.
review_content: summary_of_contributions: This paper proposes CMA-ES for post-hoc ensembling in AutoML. Therefore, first, the covariance matrix adaption evolution strategy (CMA-ES) is compared to Greedy Ensemble Selection (GES) as well as the single best model. Afterwards, CMA-ES is modified for three normalization techniques: softmax, softmax and implicit GES normalization, as well as softmax and explicit GES normalization. Those modifications are compared to each other as well as the base version of CMA-ES, stacking, and single best model. Furthermore, pseudo-discrete and sparseness of weight vectors are formalized, which is hypothesized to be properties that help to avoid overfitting.

The evaluation is done based on 10-fold cross-validations of 71 classification datasets from the AutoML benchmark of AutoGlueon, comparing the performance of balanced accuracy as well as ROC AUC both for binary and multi-class classification. The authors report that CMA-ES can prevent overfitting, and that it performs similary or better than GES.
overall_review: ## Positive

- General idea of improving ensembling to prevent overfitting due to normalization is interesting.

## Negative (see comments in other fields)

- The correctness and quality of the presentation of experimental results is low and needs improvements. Furthermore, the conclusions need to be drawn with care.
- The clarity of the paper as well as motivation and precise descriptions needs to be improved.
- The contribution and novelty of the paper is very limited.

review_summary: The paper covers an interesting idea, but unfortunately not carefully written and analyzed. Especially the presentation of the experimental results is missing important information. Overall, I am wondering about the novelty of this paper due to the limited contribution. Additionally, I was wondering why AutoSklearn2 has not been used for comparison.
actions_required_to_increase_overall_recommendation: All in all the negative aspects described under correctness and clarity need to be addressed to convince me to raise my score. Additionally, I am interested in an argument why AutoSklearn2 has not been used instead of AutoSklearn(1)?
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper studies the problem of post-hoc ensembling in AutoML systems, and proposes schemes to improve upon the widely used Greedy Ensemble Selection or GES scheme. To this end, this paper considers the use of CMA-ES for ensemble selection in place of GES by posing the ensemble selection as a gradient-free numerical unconstrained optimization of the ensemble weights using validation data.

The paper empirically evaluates this use of CMA-ES for ensemble selection on AutoML classification problems, with the results highlighting that CMA-ES is able to improve upon GES when considering the balanced accuracy metric, but falls behind GES when considering the ROC-AUC metric. Based on these results, the paper then motivates the need for constraints on the ensemble weights during the numerical optimization with CMA-ES, and presents three ways of enforcing such constraints. The empirical evaluation of these constraint enforcement schemes demonstrates that CMA-ES with such constraints improves upon vanilla CMA-ES, and is able to match GES for the ROC-AUC metric. However, these constraints lead to performance degradation with the balanced accuracy metric.

overall_review: ### Positive Aspects

Post-hoc ensembling is a widely used technique in many AutoML toolkits. Thus, any significant improvement to this ensembling scheme can have significant practical impact. This paper demonstrates that CMA-ES is able to significantly improve over GES on the AMLB when considering the balanced-accuracy benchmark.


The proposed method is clearly motivated and thoroughly evaluated with the AutoML benchmark (AMLB) with 71 classification datasets with two classification metrics. The results are presented with statistical significance tests, making it suitable for drawing meaningful conclusions from the empirical evaluations. Such an elaborate evaluation is critical when addressing fundamental aspects of AutoML such as post-hoc ensembling.

The paper does a great job at highlighting the desirable properties of GES and how to incorporate such properties as constraints with CMA-ES. This was very clearly articulated.



### Negative Aspects

I have detailed my main technical concerns with this paper in the **Technical Quality and Correctness** section of the review. Among those, the main concern is the fact that the modifications made to CMA-ES to improve performance with ROC-AUC metric leads to degradation in the performance with the balanced-accuracy metric. This gives the impression that we are not clearly understanding why we are seeing improvements or degradation over GES, and that there is some implicit aspect we are missing. Some of the other technical issues such as (i) small number of base models, or (ii) the use of AutoGluon base models, or (iii) the thresholding effect with balanced accuracy, or (iv) the fact that SingleBest performs competitively to GES with balanced accuracy might all be related to the main concern that CMA-ES enhancements for ROC-AUC hurt the balanced accuracy performance.
It is not clear if there is any fundamental improvement that CMA-ES provides over GES in a metric-agnostic way. One fundamental aspect could have been that CMA-ES allows for negative weights, while GES restricts to non-negative weights. But the CMA-ES normalizations remove this distinguishing feature, implicitly implying that the negative weights hurt.
Another related issue is that, if the CMA-ES needs to be configured differently for different metrics to get improvements over GES (or even match it), then its usability is probably going to be somewhat limited relative to GES. Would this mean that if we consider other metrics (such as F1-score, or area under the precision-recall curve), we would again have to reconfigure CMA-ES ensembling?

review_summary: The paper is studying a very important problem and provides an elaborate evaluation worthy of current AutoML research. However, I feel that there are certain fundamental limitations (as described in detail) which leads me to recommend a rejection.

actions_required_to_increase_overall_recommendation: I am happy to increase my score if the authors (or other reviewers) can point out how my concerns regarding the fundamental issues with the proposed idea is not justified or if it is a result of my misunderstanding of the idea.

Alternately, if we can understand some fundamental reasons why certain ensembling strategies or weighing schemes help with certain metrics but can hurt other metrics, it would be a valueable contribution as it will improve the understanding of the community. But this might be out of scope within a review cycle.
review_title: null

Decision: Accept(Main Track)