Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 8: Accept: Technically sound paper with major impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper combines bias mitigators and ensembling methods, and shows that ensembling reduces the volatility of the mitigators. They present a flow chart for choosing a bias mitigator and ensemble combination to use.
overall_review: The paper provides multiple contributions: a flow chart for choosing ensemble/mitigator combinations, empirical evidence for the ensembles helping, a library for combining ensembles and mitigators, and some analysis of the quality of the flow chart. The results could be made more interpretable by presenting them as plots, but overall I think it is a useful paper.
review_summary: The paper compares a number of bias mitigators and ensembling methods, and probably most usefully provides a flow chart with suggestions for what combination to use. This could be very useful, though the results could be presented in a more easy-to-digest manner.
actions_required_to_increase_overall_recommendation: Show results presented as tables as figures, to make them easier to interpret, and show that the results compared to random guessing and the best-performing combination.
review_title: null

final_score: rating: 5: Borderline Leaning Reject: Technically sound paper where reasons to reject nonetheless outweigh reasons to accept. Please use sparingly.
review_content: summary_of_contributions: The paper explores the combination of multiple bias mitigators and ensembles to improve algorithmic fairness and predictive performance. The authors developed an open-source library and guidance diagram for the composition of mitigators, ensembles, and hyperparameters, and conducted experiments with the configuration space on various datasets.
overall_review: The paper demonstrates technical rigor and holds the potential to have impact for "fairness in machine learning" while incorporating some elements of AutoML. For readers unfamiliar with the terminology, the paper may be challenging to comprehend. It is recommended to introduce key terms early in the paper, improve the formatting of tables to enhance readability, and compare with relevant existing research.
review_summary: To me it is uncertain whether the AutoML Conference is a suitable venue for publishing this work.
actions_required_to_increase_overall_recommendation: highlight the importance and potential impact of this work for the field of AutoML
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: The article discusses the challenge of maintaining algorithmic fairness in machine learning models when using bias mitigators, as their effectiveness can vary across different data splits. The authors explore the use of ensemble learning as a potential solution but note that it is unclear how to best combine ensembles with mitigators to balance fairness and predictive performance. They present an open-source library that allows for the modular composition of mitigators and ensembles along with hyperparameters, which they use to explore the configurations of 13 datasets. The authors provide their insights in the form of a guidance diagram that can serve as a starting point for practitioners. They also demonstrate that their solutions perform similarly to those generated by automatic combined algorithm selection and hyperparameter tuning (CASH) on many datasets.
overall_review: Addressing an important challenge: The paper addresses an important challenge in maintaining algorithmic fairness in machine learning models, which is a crucial issue in the field of AI.

Open-source library: The paper presents an open-source library that enables the modular composition of mitigators and ensembles along with hyperparameters, which can be useful for practitioners in the field.

Empirical exploration: The authors explore the configurations of 13 datasets empirically, which can provide insights into the effectiveness of their approach.

Guidance diagram: The authors provide their insights in the form of a guidance diagram that can serve as a starting point for practitioners, which can be helpful for those who are new to the field of AutoML.

this work adopts grid search to find the best mitigation configs, which would be inefficient when the search space is large
review_summary: The open-source library and guidance diagram provided by the authors may be useful for practitioners in the field of AutoML who are interested in improving the fairness and accuracy of their machine learning models.
actions_required_to_increase_overall_recommendation: Please provide the time and memory usage when fitting models and discuss more about the extra overhead
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound paper where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: In this study, authors explore the trade-off between the fairness and performance of different models through using 13 data-sets, eight mitigators and four ensembles. They also, suggest a guidance diagram to address this trade-off based on insights stemmed from this exploration.


overall_review: This study used a large number of data-sets and configurations to explore the trade-offs between fairness and performance which is an important topic in the community. 

However, I have some minor issues that I did not quite figure out. 

1. What was the reasoning behind 8000 rows/samples in guidance diagram? How would that be helpful or beneficial to take such split?

2. For Table 6, authors mention that "while the guidance diagram rarely recommends the exact same con- figuration as that found by Hyperopt, it recommends one with similar performance". However, roughly in half of cases(across all metrics) they have <0.1 differences and others are higher than that. Therefore, that claim is not valid. Also, is there any explanation why almost all cases of guidance F1's DI has >0.1 difference with Hyperopt suggestion(in some of them the difference is significant)?
review_summary: I believe this paper explores one of the most important trade-offs across classical machine learning that would be beneficial for AutoML community as well. With addressing the formatting problem mentioned in clarity section, I believe it has a chance to be accepted.
actions_required_to_increase_overall_recommendation: Change the captions of tables and also try to fit them properly.
review_title: null

Decision: Accept(Main Track)