Venue: AutoML 2022 Track Main

Reviews:

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: N/A for reproducibility reviewers
overall_review: N/A for reproducibility reviewers
review_summary: N/A for reproducibility reviewers
review_title: null

final_score: rating: 2: Reject, not good enough
review_content: summary_of_contributions: The authors present LassoBench which is a benchmark suite (fully available on github) tailored for weighted LASSO regression.
overall_review: They authors propose a new benchmark for high-dimensional hyperparameter optimisation; LassoBench. They suggest this is the first benchmark suite tailored for weighted LASSO regression. The main thrust of this review is that there is no novelty in this paper and even so, it is not suitable for AutoML. Certainly HPO is listed as a topic of interest, but benchmarks are not innovative enough to fit that bill as methodological innovation is sought. This being said, benchmarks are useful and there are many venus where the unveiling of LassoBench would be more impactful.
review_summary: See 'Overall Review'.
review_title: null

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: The authors propose LassoBench, a benchmark that includes both synthetic and real-world data that represents a diverse sampling of the Lasso problems out there. 

The general idea is that standard Lasso assumes a single regularization hyperparameter, but with recent advances in high dimensional HPO, the authors believe that one hyperparameter per feature is doable in a reasonable amount of time. So a second contribution of the paper is a demonstration of different HPO algorithms on this high dimensional variant of lasso (and also it examines standard CV / grid type methods)


overall_review: It's really hard to judge this paper, simply because I don't know how important this benchmark is for the AutoML community. 

I feel like some of the benchmarks are kind of.... easy? In figure 1, I don't see much variation in many of the methods. Perhaps that is because there is only one run, but I think increasing the size of the underlying datasets will make the problems a little more challenging. This of course increases the runtime, but the authors could consider releasing a surrogate (e.g., a random forest or something with fast inference) as a coarse proxy of the objective. 

I think one of the really interesting take-aways is that none of the methods was a clear winner. I think this suggests we as a community ought to recognize that each method has it's own strengths and it's own weaknesses, and be honest about what the trade-offs are (which requires a little more analysis). For example, I see that CMA-ES (which people always beat down on I feel like) does admirably for the most part. 

There's no point in horse racing against all the new stuff out there, and as this is the first AutoML conference ever, it would be really helpful of the authors to point this out for our community's sake. This would set a constructive tone and set realistic expectations for future submissions. 
review_summary: I see no reason to reject the paper; it suggests a benchmark for a very important class of methods, and I didn't find such a benchmark in the literature. The benchmark itself isn't necessarily super important for the AutoML community, but it's not any worse than AutoML benchmarks out there, and will certainly add a little more diversity to the problems that people run. 

My main criticisms are 
* Easy-ish problems, leading to not-so interesting optimization behavior... I'm worried this isn't going to particularly stress any HPO algorithm. 
* Could release a surrogate or lookup table.
review_title: null

final_score: rating: 3: Marginally below the acceptance threshold (use sparsely)
review_content: summary_of_contributions: The paper proposes a benchmark suite for high dimensional (expensive, black-box) optimization based on applications of the weighted Lasso algorithm.
As such it proposes several benchmark functions for general high dimensional benchmarks.
Benchmarks are generally real-valued and contain 60 to ~ 20k hyperparameters,
It furthermore provides a benchmark of several state-of-the-art methods on the proposed benchmark set.


overall_review: While benchmarks included in LassoBench might not be representative of the broader set of high-dimensional problems encountered in practice (axis alignment, no interactions, ...) they might nonetheless provide relevant testbeds for the specialized case. 
A qualitative comparison to the more common problem might therefore provide interesting insights for the user.

I like the fact, that multi-fidelity evaluations are possible. 
This is something that might be genuinely novel (to my knowledge) in this context.

Several important details that might allow better judging the validity of in the manuscript are missing (that would have likely be available from the accompanying experiment code): Exact computation of presented  numbers 


Other points:

How much time is saved when using low-fidelity approximations. This is unclear to me from the manuscript and might enable the reader to judge whether multi-fidelity actually makes sense.

It would perhaps be beneficial to describe (minimum) system requirements (in terms of required memory as this might be the limiting factor ?) 
I find the lack of code to reproduce the results problematic - Implementation details or the computation of values reported in tables need to be accessible if users want to extend results, especially given that raw results are also not available.

Implementation
It would be nice if the available fidelities are documented / available from the API
The API in its current format does not seem to be easily extensible without touching the initialisation method of the main class.
I am not sure why the code does not make use of inheritance and decouples data from evaluation logic. This could lead to potential errors as some functions seem just copy & pasted.
Minor: 
- How was the best guess for d_low for ALEBO and HeSBO selected? (L 271)?

review_summary: The paper is generally sound and well written, I could not detect any major flaws besides minor aspects and omissions discussed above.
In general, the manuscript does not make too many contributions. While several interesting aspects are included, I find the novelty and depth of contributions mediocre.
Furthermore, the lack of available code hurts reproducibility.
review_title: null

final_score: rating: 2: Reject, not good enough
review_content: summary_of_contributions: This paper suggests a benchmark suite for a high-dimensional hyperparameter optimization via weighted LASSO.  In particular, this project, dubbed LassoBench, consists of benchmarks for both well-controlled synthetic setups, i.e., the number of samples, noise level, ambient and effective dimensionalities, and real-world datasets.  The authors demonstrate the empirical results for their benchmark suite.
overall_review: This paper provides an interesting benchmark suite on high-dimensional hyperparameter optimization by solving a weighted LASSO algorithm.  Moreover, this benchmark is highly relevant to the AI community as well as the AutoML community.  However, I am a bit worried that the definition of high-dimensional hyperparameter optimization is ambiguous.  In my understanding, the definition of high-dimensional hyperparameter optimization is a problem with a large amount of hyperparameters.  However, high-dimensional hyperparameter optimization does not imply a problem itself.  I think that an objective function defined on a high-dimensional domain is "a problem we want to solve", and "hyperparameter optimization" is one of methodologies to optimize the objective function of interest.  Therefore, my main thought is that there is no difference between the problem solved using any optimization approaches and the problem solved in hyperparameter optimization.  For this reason, this benchmark does not need to be limited to hyperparameter optimization and moreover any high-dimensional problems can be solved by some of hyperparameter optimization strategies.  Eventually, the contribution of LassoBench is unclear for these reasons.

On the other hand, I do not agree with the statement "Potentially, CMA-ES can be initialized with the best-found solutions from the Lasso-based baselines." (in Line 348).  Applying both CMA-ES and the LASSO baseline might be redundant.  The initialization of CMA-ES does not tend to be ineffective, though.

Additionally, it is a minor issue, but in Line 55 state-of-art should be state-of-the-art and in Line 353 the sentence "LassoBench likely to ..." should be "LassoBench is likely to ...".
review_summary: This project is open-source and highly relevant to the AutoML community.  However, the contribution of this paper is limited due to ambiguous problem definition and the unclear goal of the LassoBench project.
review_title: null

Meta-review: metareview: I want to first emphasize that I disagree with a number of comments made by the reviewers, in particular that "benchmarks are not innovative enough to fit that bill as methodological innovation is sought." Many papers in AutoML (e.g., NasBench) have focused on introducing benchmarks across major machine learning venues like NeurIPS, ICLR, etc. Indeed this reviewer also comments that "This being said, benchmarks are useful and there are many venus where the unveiling of LassoBench would be more impactful." Surely, if we are to believe that the former claim is untrue, then this latter point is in fact arguing for acceptance.

I think that ultimately this paper boils down to a few pros and cons. The benchmark suite introduced appears to be a series of quite challenging problems that successfully causes a variety of quite strong hyperparameter optimization approaches to fail or at least perform differently. I think this is quite positive, as finding problems that are sufficiently challenging to develop new methods on but are not extremely difficult can be quite challenging.

On the other hand, there are two obvious drawbacks to this benchmark suite. First, I think weighted lasso  may not generate the same level of excitement as neural architecture search might. To be clear, lasso is absolutely one of the foundational methods used in machine learning, but it does not quite generate the same level of frenzied activity in the community. As a result, I think this benchmark suite may serve less as a suite of tasks that we truly want to make progress on because the problems themselves are useful to solve, but rather serve as challenging optimization problems that would allow us to measure progress in optimization performance. Second, many of the tasks included in the benchmark suite are *extremely* high dimensional -- much higher dimensional that was actually focused on in research high dimensional Bayesian optimization literature. 

Ultimately, despite generally negative sentiment about the potential impact of the benchmark, I think it's a potentially solid set of optimization benchmark functions that are potentially less likely to be picked up by the community as a whole. One of the main pieces of value I see here is that in recent years, it has become relatively trendy to use linear policy learning on reinforcement learning problems as benchmark optimization tasks for high dimensional Bayesian optimization (e.g., Swimmer/Hopper/Rover), so having very high dimensional AutoML examples could be quite useful.
confidence: 3: The area chair is somewhat confident
Decision: Accept