Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 8: Accept: Technically sound paper with major impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: While existing hyperparameter optimization (HPO) methods efficiently identify well-performing hyperparameter configurations, they often lack transparency and insights.  In this paper, the authors propose a  method that uses symbolic regression on meta-data obtained through Bayesian optimization (BO) during HPO. Symbolic regression allows them to get explicit formulas that quantify the relationship between hyperparameter values and model performance. This approach makes the HPO process more explainable and human-centered. 

Since symbolic regression directly to meta-data collected during HPO can be influenced by the sampling bias introduced by BO. The authors propose to fit the symbolic regression on the surrogate model trained during BO, which can approximate the true underlying loss landscape. 

overall_review: Pro

The state of the art is meticulously presented, offering a comprehensive overview of relevant literature in the field.

Despite being extensive and technical, the background information is clearly articulated, making it accessible to a wide range of readers.

The proposed method is intriguing and offers several opportunities for future refinement and improvement.

The limitations of the approach are transparently highlighted, such as its focus only on numerical parameters.

The experiments use an up-to-date tool (SMAC), which enhances the reliability and validity of the study.

A critical issue, the trade-off between faithfulness and interpretability, is thoroughly explored, providing a thought-provoking discussion on this topic.


Cons 

A more detailed description of the GP part is missing to my opinion

Considerations on landscapes analysis are still limited

review_summary: The main purpose of the paper was to make the HPO process more human-centered by proposing a practical workflow for practitioners to gain insights into the underlying HPO problem. The proposed approach uses symbolic regression to derive a closed-form expression that captures the relationship between hyperparameter values and their performance in an interpretable manner. The authors point out that simply learning a symbolic model on meta-data collected during the HPO process can be biased. To address this issue, they have proposed learning on randomly sampled configurations whose performance is approximated by the HPO process' surrogate model. The experimental evaluation demonstrates that a balance between the interpretability and faithfulness of the symbolic model can be achieved by systematically setting a parsimony hyperparameter.

This is clearly a relevant and good-quality contribution to AutoML

actions_required_to_increase_overall_recommendation: No specific suggestion
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper suggests an approach to make the black-box HPO process more transparent and explainable. In order to do so, the meta-data generated by a Bayesian Optimization based HPO tool (SMAC) is used as training data for a symbolic regressor. The goal of using symbolic regression is to learn the relationship between the hyperparameters and the loss value. By looking at the explicit formula discovered by symbolic regression, insights about the loss landscape of the HPO problem can be gained. However, the meta-data generated by BO-HPO tools is biased towards certain regions of the loss landscape. To avoid this, the meta-data of randomly sampled configurations is used by leveraging the Gaussian Process model learned during the BO process to approximate the loss values of randomly sampled configurations. Experimental evaluations of the HP-Bench benchmark show this approach to be favorable. 
overall_review: Pros:

1. The idea to use symbolic regression for better explainability of the HPO process is a computationally cheap method to generate inisghts and could thus be widely adopted. 
2. The proposed workflow is technically sound and straight-forward to apply. 
3. Evaluations on the HPO-Bench show the approach to be able to accurately approximate the true cost values, thus the methods seems to working well.

Cons (Major):

1. The GP baseline in Table 1 is the only baseline that the approach is compared to. What about other “simple” regression techniques? Is it obvious that linear regression or logistic regression (that are both interpretable in some sense as well) would produce worse results?
2. The main results of the experiments, namely the formulas in Section 5.2 are not explained at all. How can these formulas help practitioners of HPO to understand the loss landscape better? What do the factors (e.g. 0.078) mean? 

Cons (Minor): 

1. In the experimental evaluations just two hyperparameters for each type of machine learning model are analyzed. This is a very small subset of available (numerical) hyperparameters
review_summary: Overall, the idea and the proposed workflow with symbolic regression for explainability seem to be novel and well-suited for the problem. But adding comparisons to additional baselines is necessary to assess its real impact and a more detailed explanation of the final results is absolutely crucial for practitioners to use this approach. 

actions_required_to_increase_overall_recommendation: See review summary.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper focuses on introducing explanability and interpretability to the very critical but opaque hyperparameter optimization or HPO process. To this end, the paper utilizes the symbolic regression framework to model the hyperparameter-loss relationship utilizing the meta-data obtained from any HPO execution. The paper highlights the common tradeoff in symbolic regression between expression fidelity (the ability of learned expression to fit the data) and the expression complexity. Furthermore, the paper highlights a challenge in applying the symbolic regression framework to meta-data from a Bayesian Optimization based HPO scheme, and presents the advantages and disadvantages of two ways to mitigating this challenge.

overall_review: As I have detailed in the **Technical Quality and Correctness** section, the main strengths of this paper are the well-motivated problem and a natural, intuitive solution.

However, as detailed in the same section, one of the main concerns with the proposed idea is that it is not clear how much interpretability a simple expression for the (surrogate or actual) loss surface brings to the HPO process given the interpretability-fidelity tradeoff and the instability of the interpreted dependencies.
The choice of small HPO problems (problems with just 2 hyperparameters) does not appropriately highlight the advantage of utilizing symbolic regression over existing explanability schemes based on partial dependency plots.

review_summary: Based on the above review, the proposed idea seems an interesting direction, but in my opinion, the current conclusions and explanations seem limited or not well-supported. Hence, I recommend a rejection.

actions_required_to_increase_overall_recommendation: I am happy to increase my score if the authors (or other reviewers) can point out how my concerns regarding the fundamental issues with the proposed idea is not justified or if it is a result of my misunderstanding of the idea or results.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper introduces a new approach to make the HPO process more explainable via applying symbolic regression to the metadata collected with BO. To this end, the authors develop the inference of a closed-form expression that captures the hyperparameter values and its performance with controllable tradeoffs. Empirical experiments show that the proposed method and the systematic approach is able to faithfully model the HPO loss landscape with the tradeoff between faithfulness and interpretability controllable by the practitioners.
overall_review: Overall the idea of this paper is quite interesting. The authors first observe the feature gap that hinders wider adoption of the existing HPO tools. Then, the authors propose simple yet effective approaches to provide interpretability without infeasible overhead. Further, the authors do an extensive evaluation to illustrate the effectiveness of their approach, namely how well the symbolic models provide explanation and meanwhile accurately to capture the dynamics. The downside of this work is that it is unclear how much contribution is original to the authors as opposed to just applying and trying a combination of existing algorithms and tune the approach to work for certain settings.
review_summary: Simple yet effective approach. The paper is well written. Clarification is needed to further explain the technical contribution and to distinguish from directly applying related work.
actions_required_to_increase_overall_recommendation: Further explain the technical contribution and to distinguish from directly applying related work.
review_title: null

Decision: Accept(Main Track)