Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 5: Borderline Leaning Reject: Technically sound paper where reasons to reject nonetheless outweigh reasons to accept. Please use sparingly.
review_content: summary_of_contributions: This paper introduces two new population-based ensemble selection methods in post hoc optimization(QO-ES and QDO-ES) for AutoML, a version of ensemble selection where the search of optimism is not based on greedy search but based on population optimization. The authors present a background of ensemble learning and the motivations to pursue this work on using population-based methods instead of greedy methods. The new approach was explained in section 4 and provides guidance for each step of the proposed method.
overall_review: The paper has several notable strengths:
     The authors tackle a practical use case and propose a straightforward solution that is effective and efficient. The proposed solution addresses the problem of post hoc ensemble selection methods in AutoML.
     The paper is well-written and in an easily understandable manner.
     The results are well-visualized and straightforward to interpret.

However, there are some negative aspects to the paper:
     The proposed approach is not well presented
     It is not very clear how the method could influence on the decision making part in AutoML
     The results representation /explanation is a bit complicated to follow and need to be improved.
     The approach need to be tested on more benchmarks to make intuitive conclusions.
review_summary: The suggested problem setting is intriguing and offers a fresh perspective on post hoc ensemble selection. However, the suggested strategy is rather straightforward, and the authors offer no firm recommendations. replacing a greedy search with a population-based method to predict higher performance but still not conclusive to make better decisions for AutoML.
actions_required_to_increase_overall_recommendation: See overall review and adress comments.
The paper needs to be reorganized to show better the potential of the proposed approach.
review_title: null

final_score: rating: 8: Accept: Technically sound paper with major impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper studies post hoc ensemble election based on a population-based quality and diversity optimization for AutoML. The experimental results compared against greedy ensemble selection. The study found that diversity is one of the key factors to achieve a better performance for ensemble models. 
overall_review: It is an interesting paper for the AutoML community, but further discussion and explanation of the evaluation and the results are needed to provide a comprehensive understanding.
review_summary: The paper is well-written and easy to follow but lacks detail on experiment setup and evaluations that may impact comprehensiveness of the manuscript's findings.
actions_required_to_increase_overall_recommendation: Answering the above questions and revising the paper accordingly would be beneficial for the AutoML community.
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: 1. Using Quality-Diversity (QD) algorithms for Ensemble Selection (ES) problems. 

2. Experimental results show the advantages of QDO-ES and QO-ES. 
overall_review: Pros
1. The application of QD algorithms into ensemble selection is interesting and promising.

Cons

1. The experiment part is not solid enough. Please refer to "Actions Required To Increase Overall Recommendation".
review_summary: This paper apply QD algorithms into ensemble selection problems. However, the experiment part should be further improved.
actions_required_to_increase_overall_recommendation: 1. Abalation studies, including the operators and the hyper-parameters.

2. More types of experiments are needed.

3. The verification (or visualization) of the diversity of obtained solutions should be discussed.

4. More related methods should be compared, such as [1] 

[1] DivBO: Diversity-aware CASH for Ensemble Learning.
review_title: null

final_score: rating: 8: Accept: Technically sound paper with major impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper proposes two population based methods for post hoc ensemble selection in AutoML: QO-ES (quality optimization) and QDO-ES (quality diversity optimization). The general idea is to investigate whether the behavioural diversity of a population of ensembles may be beneficial for the performance. The methods are compared to greedy ensemble selection (GES) as well as the single best model. The authors claim that QO-ES and QDO-ES often outperform GES, as well as that behavioural diversity can be beneficial for post hoc ensembling. 

overall_review: ## Positive

- Structure of the paper
- Most descriptions of methods and evaluation protocols
- Interesting idea of behavioural diversity

## Negative (see comments in other fields)

- Results and conclusions thereof are oversold
- Mathematical presentation could be improved

review_summary: The paper proposes an interesting alternative to GES, although it is oversold in my opinion. It is well structured and described in most parts of the paper. Nevertheless, especially the results and discussions thereof could be improved and clarified. Additionally, I was wondering why only a single AutoML tool has been used for evaluation. Furthermore, in the beginning of the paper, I assumed that diversity would play a major role in this paper and would be further analyzed, but it turned out diversity was not so present.
actions_required_to_increase_overall_recommendation: All in all, the negative aspects described under correctness and clarity need to be addressed to convince me to raise my score.
review_title: null

Decision: Accept(Main Track)