Venue: AutoML 2023 ABCD_Track

Reviews:

final_score: rating: 4: Weak Reject: For instance, a submission with minor technical flaws, and limited impact.
review_content: summary_of_contributions: This paper introduce forester, a new library to perform tree-based AutoML on tabular data in R. It has a simple API that invokes the 4 steps in the pipeline in one call: (1) basic sanity checks on the data, (2) data preprocessing, (3) training and tuning of the model, and (4) performance evaluation. forester also comes with a report generator to provide users with more insights. At the end of the paper, the authors perform a comparison of forester with the H2O (another AutoML package in R) on a few binary classification dataset and show that forester is capable of building strong models.
overall_review: Pros:
- The API of forester is very intuitive and simple to use
- It comes with a nice report generator
- It runs relatively well on the 8 binary classification datasets, comparing to H2O

Cons:
- forester is too limited in terms on the type of tasks, data type, and only support tree-based models
- More experiments are needed to to validate it's performance (ideally, more AutoML libraries should be included in the comparison as well)
review_summary: The forester library is quite neat on the narrow type of task that it supports (binary classification and regression on tabular data), but it is a bit lacking in novelty comparing to existing AutoML library in R.
actions_required_to_increase_overall_recommendation: It would be nice if the author can:
- Explain what's novel in their forester work (H2O also supports wide range of metrics, so "we can compare models using the most commonly used metrics" isn't exactly novel
- See my other comments on the technical section
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound submission where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: The paper presents an R package "forester",  an AutoML package for training tree-based models on tabular data. The current implementation supports regression and binary classification. 
overall_review: Positive: Nice work, especially focusing on AutoML in R programming language. This is an important step for bringing AutoML to groups that are more familiar with biostatistics and now involving ML in their experiments.

Negative: More evaluation should be performed (more benchmarks involved), especially the regression task since the package has been reported also for solving regression tasks. Can you also estimate the impact of each step and the method selected there on the final results obtained by the pipeline, just to estimate the contribution of each step in the modeling process?
review_summary: Good work on developing AutoML methods that will be available for R users. 
More comparisons should be performed on more benchmarks in R.

Even if it is not fair, can we see what the results will be if we do this with some tool available in Python?
actions_required_to_increase_overall_recommendation: Address the issues I mention in my previous points.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact, with perhaps some minor flaws.
review_content: summary_of_contributions: 1. The paper presents an open-source AutoML package called "forester" implemented in R language.
2. The package is designed to train tree-based models on tabular data for regression and binary classification tasks.
3. The motivation for developing this package is to address the limited R solutions available for AutoML, which have high entry levels and are not accessible to everyone. The "forester" package is easy to use and accessible to users with varying degrees of ML proficiency.
4. It automates various steps inside the ML pipeline, such as handling unprocessed datasets, feature engineering, and hyperparameter tuning using Bayesian optimization.
5. It provides additional functions allowing the user to save the models, create explanations, and a detailed report describing the learning process and the developed models. The report includes the ranked list of models, a comparison of models, and explanations for the best one.
6. Furthermore, experimentation with the "forester" package indicates that it can create models that are competitive with the existing solutions.
overall_review: The "forester" package presented in the paper offers a useful contribution to the field of AutoML, especially for those who prefer to work with the R language. The package is designed to train tree-based models on tabular data for regression and binary classification tasks. It is easy to use and accessible to users with varying degrees of ML proficiency, which can save both time and effort. It automates various steps inside the ML pipeline, such as handling unprocessed datasets, feature engineering, and hyperparameter tuning using Bayesian optimization. It provides additional functions allowing the user to save the models, create explanations, and a detailed report describing the learning process and the developed models. To evaluate the performance of the package, the authors have compared it to the H2O framework on binary classification tasks from the OpenML-CC18 benchmark. The results show that the "forester" creates models that are competitive with the existing solutions.

Areas to further improve the paper:
1. The authors have tested the package for the binary classification tasks and left the regression tasks without a thorough evaluation. It would be beneficial if they could also provide a detailed evaluation of the regression tasks.
2. For the performance evaluation, the authors have only compared the "forester" package to the H2O framework, which may not be comprehensive enough and representative of all available AutoML solutions. It would be helpful if they could provide more detailed comparisons with other AutoML solutions to demonstrate how their approach compares in terms of performance and ease of use.
3. Although they have briefly discussed the potential limitations and drawbacks towards the end of the paper, the paper could benefit from more detailed insights into the limitations of their approach and potential future work to address these limitations, such as the package's applicability to other types of data or tasks.
review_summary: The paper presents a useful contribution to the field of AutoML, especially for those who prefer to work with the R language. The package is designed to train tree-based models on tabular data for regression and binary classification tasks. It is easy to use and accessible to users with varying degrees of ML proficiency. The package automates various steps in the ML pipeline and provides additional functions allowing the user to save models, create explanations, and a detailed training report. However, the paper could benefit from some improvements, such as providing detailed evaluation results of the regression tasks, comprehensive comparisons with more AutoML solutions besides the H2O framework, and a more detailed discussion of the potential limitations and drawbacks. Thus, I recommend a weak acceptance of this paper for the AutoML 2023 ABCD Track.
actions_required_to_increase_overall_recommendation: Authors can take the following actions:
1. The authors have tested the package for the binary classification tasks and left the regression tasks without a thorough evaluation. It would be beneficial if they could also provide a detailed evaluation of the regression tasks.
2. For the performance evaluation, the authors have only compared the "forester" package to the H2O framework, which may not be comprehensive enough and representative of all available AutoML solutions. It would be helpful if they could provide more detailed comparisons with other AutoML solutions to demonstrate how their approach compares in terms of performance and ease of use.
3. Although they have briefly discussed the potential limitations and drawbacks towards the end of the paper, the paper could benefit from more detailed insights into the limitations of their approach and potential future work to address these limitations, such as the package's applicability to other types of data or tasks.
review_title: null

final_score: rating: 3: Reject: For instance, a submission with technical flaws and limited impact.
review_content: summary_of_contributions: The authors propose forester, an open source AutoML system written in R that performs data checks, data preprocessing, model training, and model evaluation in an easy to use interface. forester uses tree models including GBMs, along with hyperparameter tuning. A benchmark on 8 small datasets shows that forester obtains similar performance to H2O AutoML, winning on 5 out of 8 datasets.
overall_review: While forester does have positive aspects such as its simple design, automated data preprocessing, friendly API, easy to understand result reporting, and open-source availability, these aspects are in general available in numerous widely adopted, powerful, and easy to use open-source AutoML frameworks (ex: H2O AutoML, Auto-sklearn, FLAML, AutoGluon). The benefit of being available in R is an interesting aspect, however H2O AutoML is also available in R, and the experiments conducted are insufficient to meaningfully compare the systems. Further, availability in R is questionable to its impact, and the repository on GitHub lacks indication of significant user adoption compared to the aforementioned Python-based AutoML systems. The main positive aspect in my mind is the data cleaning, preprocessing, and result reporting, however these aspects are not explored via ablation studies, and are not sufficient in and of themselves to make a compelling AutoML system. Beyond the above, the experiments conducted appear to have correctness issues as I describe in detail earlier in the review.
review_summary: I am inclined to reject the paper due to the lack of meaningful experiments, ablation studies, and baselines, as well as the questionable utility forester provides beyond existing AutoML systems. Finally, the experiments that were conducted appear to have major correctness issues.
actions_required_to_increase_overall_recommendation: I encourage the authors to fix the issues mentioned regarding their experiments, and expand the experiments significantly to cover many more datasets. It would benefit the relevance of the paper greatly if it were to integrate into the AutoMLBenchmark for example (https://github.com/openml/automlbenchmark).

The authors should additionally perform ablations on their key contributions to enhance their impact and trustworthiness.

While addressing the above would enhance my score, I struggle to see relevant impact potential for the work that justifies acceptance unless from a performance standpoint it compares favorably to existing systems or the ablations reveal compelling and novel results.
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound submission where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: This work introduces the forester package in the R language The package provides an easy-to-use framework for beginners and experts alike to use for AutoML using tree-based methods on tabular data.

The presentation includes discussion on the similarities and differences to two existing packages. It presents the package's main components of its AutoML pipeline as well as some additional features for saving and understanding the results.

It also includes a set of benchmarking results in classification, showing forester comparing well to another existing package.
overall_review: ### Strengths
* The proposed package covers the full AutoML pipeline, not just optimisation. 
* Provides an easy-to-use framework that can be especially useful for non-experts.
* It introduces a novel and useful data quality checking step.
* It allows for HPO using Bayesian optimisation, random search or setting default params.
* Uses the DALEX package for explaining model behaviour.
* The code repository seems popular, with 93 stars, good issue handling, and a history of several developers contributing.
* It is an open-source software package under the GPL 3 license.
* This package will allow applying AutoML to real-world applications in the R language.
* The paper is well written.

### Weaknesses
* Provides similar functionality to the tidymodels, caret, mlr3 and H2O packages, but adds extra features like data quality checking. 
* It does not support multinomial classification.
* The experimental evaluation is only performed for classification, not for regression.
* There is no evaluation of the time complexity of the package, i.e. how fast it is compared to H2O when performing the same task.
review_summary: A novel framework that can be very useful for R users who are new to the AutoML process. I lean towards accepting the paper but am concerned about the package's similarities to existing frameworks as well as missing experimental evidence.
actions_required_to_increase_overall_recommendation: Actions that would increase my rating are:
* Please clarify the similarities and differences to other existing packages (especially tidymodels and caret, in addition to H2O and mlr3 which are discussed). A list or table of features available in each would be helpful.
* Results for regression benchmarking.
* Results for how fast forester is compared to e.g. H2O or another existing package.
review_title: null

Decision: Accept(Workshop)