Venue: AutoML 2023 ABCD_Track

Reviews:

final_score: rating: 8: Accept: Technically sound paper with major impact, with perhaps some minor flaws.
review_content: summary_of_contributions: N/A, Reproducibility Review.
overall_review: N/A
review_summary: N/A
actions_required_to_increase_overall_recommendation: Writing a basic `demo.py` would greatly help the user understand how the code works.
review_title: null

final_score: rating: 6: Borderline Leaning Accept: Technically sound submission where reasons to accept outweigh reasons to reject. Please use sparingly.
review_content: summary_of_contributions: The authors propose ABLATOR, a system to horizontally scale ML experiments while reducing the rate of implementation errors and false conclusions via stateful experiment design focused on persistence and robustness. ABLATOR provides actionable analytics automatically from experimental runs, and supports multi-node trial runs using Ray Core as a backend. The authors additionally use ABLATOR to evaluate 'Tabulator', a tabular transformer model, against FT-Transformer and analyze the impact of hyperparameters on the performance across 11 tabular datasets.
overall_review: This is a strong paper that proposes a useful experiment execution, tracking, and analysis suite along with relevant code for reproducibility and analysis of insights.

My biggest concern is the strange take-aways from Table 1 that seem to contradict the data, which I would like clarification on by the authors in the rebuttal. I am also a bit confused on the positioning of 'Tablator' in the paper. Is this supposed to be a novel model introduction?

Additionally, it is unclear given all these experiments what the actionable take-away is, as it appears that even with over 2000 trials on 11 datasets that no statistically significant result was observed. What part of this process is "Auto" if anything in regards to "actionable" take-aways? How many experiments would we need to run to identify actionable take-aways, and would it scale?
review_summary: I am borderline leaning accept on the condition that my concerns are addressed in the rebuttal, particularly regarding Table 1.

[Update]: While the rebuttal answered some of my more minor questions, the core questions on actionable take-aways & table 1 were not addressed sufficiently, and thus I am updating to borderline leaning reject, though this is a very close call overall as the paper has a lot of strengths.

[Update 2]: The authors have convinced me that the focus should generally lie on the utility of the tool rather than the technical rigor of the meta-analysis section, and thus I am updating my score to a 6 (borderline leaning accept).
actions_required_to_increase_overall_recommendation: I would consider raising my score assuming my concerns are addressed in the rebuttal, particularly regarding Table 1.

[Update 2]: This has been mostly addressed, and thus I updated my score
review_title: null

final_score: rating: 5: Borderline Leaning Reject: Technically sound submission where reasons to reject nonetheless outweigh reasons to accept. Please use sparingly.
review_content: summary_of_contributions: This paper introduces a stateful formulation of horizontal scaling for ablation experiments to combat common errors during development. It instantiates this formulation into the ABLATOR framework in PyTorch.

In addition to the framework, the paper gives recommendations for good practice and performs a big benchmark on tabular data.
overall_review: ### Strengths
* The framework formulation is rigorous and well-thought-through.
* It is an open-source software package released under the GPL 3 license.
* The code repository is very recent but has some following already.
* The package gives broad functionality relevant to the AutoML community.

### Weaknesses
* The codebase and documentation lack tutorials or introductory information.
* It is hard to understand what functionality this framework provides. It would help to have more visual depictions of this, both in the paper, code repository and documentation.
* The codebase has only one contributor, and few issues have been closed.
* The focus of the paper, especially the experimental section, is split between presenting the ABLATOR framework and attempting to provide novel insights. It leads to neither goal being fully achieved.
review_summary: A good framework that suffers from poor documentation and a paper that lacks focus and tries to cover too many bases.
actions_required_to_increase_overall_recommendation: My main concerns are:
* The focus of the paper is a bit unclear. I think it would benefit from fully focusing on the ABLATOR framework and showcasing its functionality and effectiveness. This might just involve clarifying better in the text how everything fits together or it might mean exploring questions like how ABLATOR performs compared to alternative approaches in terms of usability and time complexity.
* It would help to have more visual depictions of the framework's functionality and how it compares to other alternatives, both in the paper, code repository and documentation.
* The framework lacks guidance on how to use it, in the form of tutorials or examples. The documentation is very bare-bones and seems to be purely based on automatically generated information from code. Adding these in would massively increase the potential usefulness of this application.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact, with perhaps some minor flaws.
review_content: summary_of_contributions: This paper proposes a new software framework for large-scale ablation studies in machine learning pipelines, pytorch models in particular. 

overall_review: The paper provides an interesting perspective on ablation studies in a part of machine learning, and illustrate how their proposed software can help make this type of analysis easier to execute. 

The provided example use-case shows hints of its applicability, but it is somewhat limited in scope. The mixing between usecase-specific and general tool-based functionality can be confusing in places, so a clearer separation might help both show the condiderations made in the design as well as the resulting types of data for supporting the user in their analysis. 

review_summary: The paper clearly describes the challenges with executing ablation studies in the context of large pytorch-models, and proposes a software-based solution which addresses these challenges. While the clarity of the use-case can be improved, the overall paper seems to be relevant to the track. 
actions_required_to_increase_overall_recommendation: The main point I would like to see improved is the clarity of section 3, with regard to more directly separating the particularities of the use-case from the general functionality of the tool.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper introduces ABLATOR, a novel framework that facilitates conducting horizontal-scale ablation studies with ease. The framework is implemented based on the "stateful experiment design" paradigm, which is also introduced in this work. This new paradigm encapsulates the design, execution, and analysis of experiments, aiming to provide robustness against errors and facilitate hypothesis evaluation.
overall_review: Pros:
* The framework has the potential to simplify horizontal scaling of machine learning ablation experiments.
* The paper identifies common errors that can occur when horizontally scaling ablation studies and provides suggestions to avoid them.
* The framework contains enough documentation and examples on how to use it.

Cons:
* The paper does not provide a broader discussion of ABLATOR's limitations, such as whether it can be used with models of any kind (e.g. classification, regression, segmentation).
review_summary: The paper introduces ABLATOR, a novel framework designed to simplify the horizontal scaling of ablation studies. Additionally, the authors identify common errors found in horizontally scaling ablation experiments and provide suggestions to mitigate them, introducing the paradigm of "stateful experiment design" to ensure the robustness of the framework. Although the authors conducted the largest ablation study for tabular data, it would be beneficial to include experiments with other models tackling various problems and data types to showcase the versatility of ABLATOR. A more comprehensive documentation on how to use ABLATOR and how to customize it would also enhance its impact.

[Update] The authors have included a comprehensive documentation on how to use ABLATOR.
actions_required_to_increase_overall_recommendation: All of my concerns regarding the paper have been mentioned above. However, here is a summary of the suggested actions that could further enhance the quality of the work: (1) The versatility of ABLATOR should be demonstrated by performing ablation studies on more models with different data types and addressing various problems. (2) Providing a detailed documentation on how to use and extend ABLATOR, including examples in the GitHub repository, would be highly beneficial. (3) It would be beneficial to include a discussion on the compatibility of ABLATOR with existing libraries and frameworks.

[Update] The authors have addressed all my concerns during the rebuttal. Therefore, I have updated my rating accordingly.
review_title: null

final_score: rating: 7: Weak Accept: Technically sound paper with moderate-to-high impact, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper proposes Abalator, an open source framework for running and managing ablation experiments in machine learning. The framework requires two inputs, the experiment design config which describes the set of ablations and the model implementation. It outputs artifacts which are figures, visualizations comparing the performance of different parameters.
The framework is fault tolerant and persists experiment state to recover from failures. It supports PyTorch models and uses the ray library for distributed execution. The authors evaluate the system using a large ablation study involving 2300 runs. 

overall_review: Pros:

1. Addresses a practical and common problem in ML of running ablation experiments. The framework is built on top of existing tooling which makes it easily extensible. 
2. The features for fault tolerance and artifact results generation are useful. 
3. The discussion around common source of errors in ablations is insightful,


Cons:
1. Comparison with existing tools (like AutoAblation) and how Ablator is better is limited and needs more detail.
2. The description of how the ablation runs are created from config is missing. Similarly the set of ablation policies and configuration features for setting up the trial aren't discussed in detail.
3. Missing description of limitations.
4. The github page mentions Ablator can be compared with hydra for configuration and ray for HPO, but its unclear how similar or different they are and more documentation is required.

review_summary: The paper is well written, the proposed framework is useful and relevant for running ablations. More discussion around features and comparison with existing tools is required. 
actions_required_to_increase_overall_recommendation: Address the cons described in overall review
review_title: null

Decision: Accept