Venue: AutoML 2023 ABCD_Track

Reviews:

final_score: rating: 8: Accept: Technically sound paper with major impact, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper introduces AlphaD3M, a Python library that's capable of automatically constructing machine learning pipelines on a wide range of data types and learning tasks. It works on the D3M primitives, which is compatible with the entire D3M ecosystem. AlphaD3M comes with a flexible API that's accessible to both basic and expert users. It also includes a novel system to generate tasked-based CFG for each specific task, which was shown to outperform manually curated grammar in the experiment section. 
overall_review: Pros:
- AlphaD3M is a nicely documented, open-sourced, library with accessible API
- It supports a wide-range of tasks and data types
- It comes with a novel task-based grammar generation system and avoid biases from manually-define grammar, and the author performs ablation study to prove its effectiveness
- It is compatible with the D3M ecosystem


Cons:
- AlphaD3M doesn't show a great performance improvement over existing AutoML libraries
- The script to reproduce benchmark result is missing
review_summary: Overall, AlphaD3M has a potential to be a one-stop AutoML toolkit for general needs because of its capability to support a wide range of tasks. 
actions_required_to_increase_overall_recommendation: See my reviews on technical section for details
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper proposes an open-source Python library, called AlphaD3M, which is able to automatically construct pipelines for 17 different unique learning tasks by combining deep reinforcement learning and meta-learning.

The pipelines found by the AlphaD3M have been compared with other pipelines obtained by other AutoML software/libraries and comparable results have been presented. AlphaD3M is able to find solutions for 17 different learning tasks, while some other AutoML libraries are specialized for a subset of tasks. There are also other libraries such as Aika that work for all included learning tasks. 
overall_review: Positive: The paper proposes an open-source Python library, called AlphaD3M, which is able to automatically construct pipelines for 17 different unique learning tasks by combining deep reinforcement learning and meta-learning. The pipelines found by the AlphaD3M have been compared with other pipelines obtained by other AutoML software/libraries and comparable results have been presented. It is really nice written paper. I believe this is a nice contribution to the AutoML community.


Negative: Even benchmarking results are presented on D3M datasets and OpenML benchmark, maybe a statistical analysis should be performed to see the benefit of using the library against the others already available. 
review_summary: The paper is well written, with minor flows that can be explained. The part that is missing is what is the contribution of the library compared to the already existing one. The state-of-the-art is nicely summarized in a table, however, in the benchmarking, it seems that other libraries/tools are able to solve 17 or 16 learning tasks. Maybe you can estimate what is the performance gain in cases when AlphaD3M is superior to pipelines find by other tools.
actions_required_to_increase_overall_recommendation: - Estimate what is the performance gain of pipelines found by AlphaD3M against pipelines generated by other tools.

- Address the minor comments presented above.
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound submission where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: The authors present AlphaD3M, an open source Python based AutoML system that can handle a wide variety of task types with only a few lines of code. AlphaD3M incorporates reinforcement learning & meta-learning for pipeline selection trained on Marvin, a meta-corpus of 600 datasets and 2.5M pipelines. The meta-learning is informed based on task keywords which are manually declared by the user as part of the fit call. AlphaD3M splits hyperparameter tuning into a post-pipeline selection stage and leveraging existing bayesian optimization techniques to tune the final result. The authors emphasize AlphaD3M's application within user-in-the-loop scenarios, and present two benchmarks that showcase AlphaD3M's performance compared to existing AutoML systems, claiming comparable or superior performance to state-of-the-art AutoML systems.
overall_review: This is a solid paper discussing a novel and feature rich open source AutoML system that currently falls short on experimental rigor, reproducibility, and analysis, leading it to make unsupported claims on performance. In particular, it does not mention the risks or mitigation strategies of benchmarking on datasets that may have been used in meta-learning, and does not provide sufficient details to reproduce experiments nor code to do so. Refer to the above comments for details.
review_summary: I am inclined to weakly reject the paper in its current form due to the lack of experimental result breakdown and the unsupported claims of "performance that is superior or on par with those derived by other systems." via metrics such as average rank or relative error, along with the lack of code to reproduce the results of any of the experiments.

[Update post-rebuttal] I have increased my score to a 6 (borderline leaning accept) based on the author's responses. The score is contingent on the authors updating the requested tables and figures to support their performance claims in the camera-ready version.
actions_required_to_increase_overall_recommendation: The AlphaD3M system is compelling and novel, but the paper falters in the experiments section. I am open to increasing my score substantially if the experiments are given more clear definitions, exact result numbers are provided in appendix tables, and average ranks are calculated to support the performance claims of the system. I would be even more open to increasing my score if code was provided to be able to reproduce and verify the experimental results. Finally, discussion should be added on how overfitting was avoided in case any benchmark datasets were used for meta-learning.
review_title: null

final_score: rating: 8: Accept: Technically sound paper with major impact, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper introduces a new AutoML library that can handle a wider range of machine learning problems and data types compared to existing libraries. It's designed to be user-friendly, even for novices, thanks to the automatic derivation of CFG through meta-learning. Additionally, to enhance the quality of the generated pipelines, the authors include a notion of prioritization for primitives, building on previous research.
overall_review: The proposed library has the potential to facilitate further advancements in AutoML systems due to several key advantages. Firstly, all the components of the library are clearly defined and justified, making it easier to extend and develop. Secondly, it supports a broader range of ML tasks and datatypes compared to other existing systems. Additionally, the performance of the pipelines generated by the library is either comparable to or better than other systems. Furthermore, the GitLab repository provides enough examples of how to use the library for various types of problems, and the API is well-documented. Lastly, the main components of the library have been ablated to demonstrate their contribution.
review_summary: Overall, the advantages of the proposed library are clear, the analysis is comprehensive, and the paper is well-written and easy to understand. Furthermore, the paper provides scope for further improvements and extensions to the proposed library, indicating its potential for future advancements in AutoML systems.
actions_required_to_increase_overall_recommendation: 1. Include Auto-Sklearn 2.0 in the comparison to gain a more comprehensive understanding of the performance of the proposed library compared to other state-of-the-art AutoML systems.
2. Verify the claim that "AlphaD3M had the best overall performance, producing the best pipeline for 39 datasets" by running the top three systems, including AlphaD3M, only with tasks that can be solved by all of them. Then, compare the number of best pipelines generated by each method. If AlphaD3M manages to generate the highest number of best pipelines, it will ensure that its superior performance is not solely due to its support for a broader range of ML tasks.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper introduces AlphaD3M, an AutoML library that automatically synthesizes end-to-end pipelines for 17 different machine learning tasks and six data types. The main contributions include new methods for automatically deriving grammars and prioritizing primitives, which help manage the search space effectively. The system also adopts a user-in-the-loop approach with an API that allows users to explore input data and derived machine learning pipelines. The authors perform a comprehensive experimental evaluation comparing AlphaD3M to various state-of-the-art AutoML systems over different problems and datasets. The results demonstrate that AlphaD3M is effective in solving a larger number of problem types and generating pipelines with performance that is either superior or on par with those derived by other systems.
overall_review: Strengths:
1. AlphaD3M addresses a wide range of ML tasks and data types, which is a significant advantage over many existing AutoML systems that only support a limited number of tasks.
2. The automatic construction of grammars and prioritization of primitives are novel and help in handling larger search spaces.
3. Provides a thorough experimental evaluation using two dataset collections (D3M Datasets and OpenML Benchmark) and compares the performance of AlphaD3M with state-of-the-art AutoML systems.
4. Inclusion of real use cases helps demonstrate the practical utility of AlphaD3M in real-world scenarios and its ease of use.
5. The user-in-the-loop approach, supported by an API for exploring input data and derived ML pipelines, enhances usability and makes it a more practical solution for data scientists.

Weaknesses:
1. The current version of AlphaD3M relies only on the D3M primitives to build ML pipelines. Incorporating additional state-of-the-art and more recent primitives from other repositories would be beneficial in improving the system's capabilities.
2. The paper does not provide an in-depth analysis of the scalability of the system. It is unclear how the system would perform with even larger datasets or more complex problems.
3. Integration with other libraries or tools: The paper does not discuss potential integration with other state-of-the-art and more recent primitives, such as models published in HuggingFace or PyTorch Hub repositories. Including such primitives would extend the capabilities of AlphaD3M and make it more appealing to a broader audience.
4. More details on below two items would add value:
4.1 Automatic grammar construction:
a. How does the automatic grammar construction algorithm deal with complex data types and situations where relationships between features are not easily discernible?
b. How do you ensure the quality of the generated grammar, especially when dealing with noisy or incomplete data? Are there any quality metrics or validation methods applied during the grammar construction process?
4.2 Prioritization of primitives:
a. How does the system learn or update the prioritization of primitives over time, especially when new primitives are introduced or existing primitives are updated?
b. Can you elaborate on the criteria used to prioritize primitives? How are these criteria determined, and are they adaptable to different domains or problem types?
c. How does the system handle cases where multiple primitives have similar performance or impact on the pipeline? How does it determine which primitives to prioritize in such cases?

Additional questions for the authors:
1. How does AlphaD3M handle cases where the dataset has missing values or noisy data? Are there any preprocessing steps or data cleaning techniques incorporated into the system?
2. How does the system perform when handling imbalanced datasets? Are there any specific techniques applied to address this issue?
3. Can you provide more details on the execution times of AlphaD3M compared to other AutoML systems, especially when dealing with larger datasets?
4. Are there any plans to extend AlphaD3M to support additional ML tasks such as regression, time series forecasting, or reinforcement learning?
review_summary: The paper presents AlphaD3M, an AutoML library designed to synthesize end-to-end pipelines for a wide range of machine learning tasks and data types. The system introduces new methods for automatically deriving grammars and prioritizing primitives, improving its ability to handle larger search spaces. The paper provides a detailed experimental evaluation comparing AlphaD3M with other state-of-the-art AutoML systems, demonstrating its superior performance and coverage.

However, the paper could benefit from discussing potential integration with other state-of-the-art libraries and tools could expand the capabilities of AlphaD3M and make it more appealing to a broader audience. Also adding more in-depth explanations of some components, such as automatic grammar construction and prioritization of primitives would strengthen the paper.
actions_required_to_increase_overall_recommendation: Addressing questions under overall review section - please see weaknesses, additional questions.
review_title: null

Decision: Accept