Venue: AutoML 2023 ABCD_Track

Reviews:

final_score: rating: 8: Accept: Technically sound paper with major impact, with perhaps some minor flaws.
review_content: summary_of_contributions: The submitted work build upon previous research for generating benchmark functions.
While two different works (one involving the authors) have already worked on affine recombination of BBOB functions, the main contribution of this work is:
- that the location of the global optima can be placed arbitrarily
- some previous issues are alleviated by e.g. the scaling factor
- the authors highlight the existence of shortcomings in the area of exploratory landscape analysis via an algorithm selection scenario

overall_review: Positive Aspects:
- the paper is well structured
- the main contributions are clearly stated and meaningful while also contextualized with the current state/shortcomings of existing research
- the constructed experiment are sound and mostly serve the purpose of conveying a coherent message.

Negative Aspects:
- Figure 1 does not really provide evidence that a scaling factor is needed. In fact, it seems that scaling is unnecessary for the chosen example
- While the entire code base is submitted, it is not entirely reproducible
(- Many aspects of this work requires prior knowledge to fully understand the merits of this work. However, 9 single-column pages is rather harsh and probably not something the authors can really address)
review_summary: In general, I enjoyed reading the paper. Disregarding some minor issues, it is well written and clearly states its contribution. These contribution (while not groundbreaking) provide a needed opportunity to test HPO algorithm more rigorously.
Furthermore, it highlights possible shortcomings of exploratory landscape analysis features, which are not able to fully capture the landscape characteristics of these newly created problem instance. Lastly, it points out a clear avenue for future research, where the selection of representative subset of problem instances is the most prominent one.
actions_required_to_increase_overall_recommendation: Minor issues:
- p.2 "If the instance space [...]", I assume instance is referring to all problems in general. However, this is not entirely clear since two sentences before instances in terms of BBOB are mentioned.
- p.2 "insights into e.g." -> ", e.g.," since this is convention is used on p.1 as well
- p.2 $1\,000$ instead of 1000. The former is used throughout the paper
- p.2 Capitalize the term Exploratory Landscape Analysis, since you have done that also on p.1
- p.6 "pFlacco" -> "pflacco"
- p.6 "[...], reduce the original [...]" missing a word lilke "we" in front of reduce
- p.6 "To gain insight [...]" repetitive use of this phrasing, i.e., two sentences before.
- Fig.7 the x-axis labels are sometimes overlapping and it is difficult to associate them to the corresponding axis ticks.
- Fig.8 b) AOC -> AUC
- Fig.9 a) AOC -> AUC
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper presents a new version of the commonly used BBOB suite. It combines multiple existing functions to create new benchmark problems. The necessary scaling and sampling strategies are discussed before delving into experimental validation through analysis of the function landscapes, algorithm performances and selection. This provides some insights into changes in the best performing algorithms and the quality of ELA features.
overall_review: ### Strengths
* Extending a popular benchmark for black-box optimisation which can drive further progress in this field.
* Simple idea of combining existing functions into new ones.
* The evaluation of the benchmark provides some deeper insights into algorithm performance.
* Benchmarking data and tools are easily accessible

### Weaknesses
* The paper introduction is not clear. It would benefit from a simplified high-level intro to the area. Some discussion in the current intro could be moved to the related works section instead.
* The codebase for the benchmark could have clearer guidance on its usage. Tutorials, examples and more documentation in its notebook would be particularly helpful.
* The paper currently does not give clear instructions for how future work can use the benchmark, wrt standardised evaluation.
review_summary: A simple but interesting extension to a black-box optimisation benchmark. It has the potential to improve the evaluation of such approaches, but first needs to provide guidance on its usage, standardised evaluation and more documentation. The paper would also improve by providing a simple introduction to the area before delving into detail.
actions_required_to_increase_overall_recommendation: My main concerns are:
* The benchmark codebase could include more guidance on its usage, such as tutorials or better documentation.
* The paper could give more details on how the benchmark should be used by future researchers.
* The paper would benefit from a simplified high-level introduction to the area.
* Clarify how this work is relevant to the AutoML community in particular.
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound submission where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: The paper proposes a new approach called MA-BBOB for generating instances for numerical black-box optimization benchmarking. MA-BBOB allows for multiple affine combinations of the well-established BBOB functions and global optima locations, which can help fill the instance space and preserve overall patterns in algorithm performance. The authors also explore the usefulness of combining landscape features with performance data to improve optimization algorithms. The paper's main contribution is to extend a recent suggestion for generating new instances and provide a more general framework for evaluating AutoML approaches in noiseless numerical black-box optimization contexts.
overall_review: Positive aspects:
1. The paper proposes a new approach called MA-BBOB for generating instances for numerical black-box optimization benchmarking, which extends a recent suggestion for generating new instances and provides a more general framework for evaluating AutoML approaches in noiseless numerical black-box optimization contexts.
2. The authors explore the usefulness of combining landscape features with performance data to improve optimization algorithms, which could be valuable for researchers working on AutoML approaches in similar contexts.
3. The paper includes relevant background information and related work to contextualize the proposed approach and its contributions.
4. The experimental evaluations are thorough and include standard measures such as the expected running time and the number of function evaluations, as well as comparisons with other state-of-the-art methods.
5. The authors provide insights into the usefulness of their approach and potential directions for future research.

Negative aspects:
1. The introduction could be more concise and focused on the main contributions of the paper, as it currently includes some unnecessary background information.
2. Some technical terms and concepts may not be immediately clear to readers who are not familiar with numerical black-box optimization benchmarking, so additional explanations or examples could be helpful.
3. While the experimental evaluations are thorough, they focus primarily on a first analysis of randomly sampled instances rather than representative instance collections or generalization ability to other problems outside of BBOB family.
4. Although the authors discuss potential negative societal impacts in their submission checklist, they do not elaborate on this topic in the paper itself.
review_summary: Overall, I think that the paper has several positive aspects such as proposing a new approach for generating instances in numerical black-box optimization benchmarking and exploring useful combinations of landscape features with performance data to improve optimization algorithms. However, there is room for improvement in terms of clarity, technical explanations, experimental evaluations focusing on representative instance collections or generalization ability, and elaboration on potential negative societal impacts.
actions_required_to_increase_overall_recommendation: no
review_title: null

Decision: Accept