Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This work presents a software system for AutoML based on ML pipelines that:
+ Uses 3 "combinators" to construct pipelines.
+ Abstracts AutoML search using a "translation scheme".

A second part of this work focuses on user adoption, backed by usability studies.
The user study described in the supplementary material is valuable.
main_review: The contribution of this work as an AutoML system, to the automated composition of complex models, is incremental.
This work may benefit by focusing on the user study, specifically on human-model interaction.

The work is missing:
- Comparisons with other types of AutoML systems that are not pipeline oriented, such as
* AutoGluon which focuses on generating complex ensembles and stacking
* OBOE which focuses on collaborative filtering and fast recommendations.
- Computation times of the system.

The work may reference the D3M project which is composed of three tiers:
(i) TA1 ML primitives, (ii) TA2 AutoML systems, and (iii) TA3 human in the loop and user studies.
The paper is missing multiple references to work on the interface between (ii) and (iii) and systems which focus on user studies:
See https://datadrivendiscovery.org/d3m-interfaces

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: In the paper „Pipeline Combinators for Gradual AutoML“ the authors propose a machine learning library that allows for convenient definition and usage of ML pipelines. In particular, the library supports the configuration of a pipeline to be (partially) automated.
main_review: Overall, I very much enjoyed reading the paper since the structure and the language of the paper are excellent.

The ideas on which the library is based seem to be very targeted and helpful for using it in practice.
In the paper the authors give sufficient detail about their library and the specific features that distinguish their library from others such as scikit-learn. Everything is explained in much detail and the architecture and the design appears to be sound.

However, in some respects the paper is very technical (due to the nature of its contribution, of course) but it could be less of the perspective of software engineering and focus more on the machine learning / data science part.

Speaking about the machine learning part, and thus, about the users of the library, I appreciate that the authors carried out a user study. However, I have sone concerns about the study itself:

- First, it is stated that all subjects are from the same company. Guessing that the library is developed by the same company this might mean some bias towards the company‘s product. Be it due to loyalty or due to the library being implemented according to the company‘s principles and philosophies.

- Second, it is stated that all subjects of the study are experienced users of scikit-learn. All the more I was very surprised that several  subjects couldnt identify the right pipeline shape. How is „experienced“ defined? I would have expected that at least reading code should be easy.

- Third, the user base is relatively small in order to reliably draw conclusions based on the acquired data. This limitation should be mentioned somewhere. 

To sum up, I believe that the library is a valuable contribution to the community and I hope that acceptance by users will be reasonable. 
limitations_and_societal_impact: The number of subjects in the user study is rather small and homogeneous, which also means a limitation to the generalizability of the results. This should be mentioned somewhere.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a framework and library of operators for enabling so-called gradual automl. The operators share similarities and extend scikit-learn's methodology of creating ml pipelines. Operators can be nested, creating complex constructs, which can be automatically translated into automl search spaces. The hyperparameters are defined by schemas.

The library is evaluated in terms of a user study with 18 participants, testing for pipeline understanding, construction and improvement following the introduced progressive disclosure concept. The evaluation also shows empirical results on 14 OpenML classification datasets for available automl algorithms in the framework compared to standard auto-sklearn, showing competitive performance. Finally, the authors also show that different optimizer backends, such as HyperOpt, can be used in the library.
main_review: Positive factors are ...
- Operators could be an intuitive approach to modeling pipelines
- Conceptual differences to other libraries are made clear

Negative factors are ...
- Concept of gradual AutoML not fully convincing, experimental results do not fully back claims

The paper thoroughly introduces the concept of available/implemented combinators, execution modes as well as the developed schema. The design of the framework is qualitative, elegantly enabling to model complex pipelines in nested structures. 

The authors also provide a detailed related frameworks and libraries, as well as key differences in design decisions. The clarity of the framework components and differences to related works is therefore high. The originality of the framework is also sufficiently high, as it gives practical value to AutoML end-users. 

To this end, I have problems to grasp the "gradual" aspect of the approach, which is part of the key message of the paper. The concept is introduced and explained, and a mapping to individual framework functionalities is available, but the latter merely enable manipulation on different abstraction levels. It would be beneficial to clearly state the novelty here, if it is a central claim.

That said, while the paper maps different gradual AutoML stages to library functionalities, the evaluation does not attempt to compare them to frameworks other than scikit-learn. Why was no other library, as described in the related work section, chosen in addtion? scikit-learn is surely a good baseline, but comparing it to other libraries which enable easier / more interactive modeling might help to support the claims.

### Update after author response
I thank the authors for their answers to my questions for which I do not have any follow-ups. After reading the other reviews and discussions, I raised my score to a weak accept, as I agree that the paper does not need to show superior performance to other AutoML tools. I still believe that it would be beneficial for the venue to better quantify the added value of the proposed "gradual" property, e.g. in the user study. 
limitations_and_societal_impact: The authors address both conceptual limitations as well as negative societal impact. They explain shortcomings in the current framework an point out that AutoML raises CO2 omission.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a system for gradual AutoML within sklearn. The system focuses on _gradual_ automation, as opposed to total automation, so that the data scientist can incorporate their human intuitions into the AutoML process, saving time and compute resources. The paper introduces combinators that enable developers to write learning pipelines in sklearn, possibly with choices in the model architecture that the AutoML system can search over. There are only 3 combinators (pipe, and, or), their function is easy to understand, and they can be combined and nested arbitrarily. While combinators describe a space of pipelines to search over, the paper also uses an extension of JSON Schema to describe a space of hyperparameters to search over. The paper describes how the schemas can be transformed into formats that are accepted by existing AutoML algorithms. The authors also perform a user study investigating how easily their proposed system can be used, and perform empirical studies comparing how well their AutoML system performs compared to other systems on many datasets.
main_review: **Positives:**
* Important problem
* Easy-to-use system for gradual AutoML
* Very clear writing
* Well-thought-out arguments for the benefit of gradual automation and how Lale enables this

**Negatives:**
* Limited novelty
* Conclusions from the experiments are not laid out as clearly as I would like

---

This paper tackles an important problem: creating a framework in which users can use AutoML technologies easily, with fine-grained control over the search space of model architectures and hyperparameters. The proposed system appears to achieve this goal well. It is easy to understand how to use the three combinators to construct pipelines, possibly with different choices in them that give rise to a space of pipelines to search over. It is also easy to use their proposed schemas to specify a space of hyperparameters as well. Altogether, the authors’ Lale library seems like a great way to enable gradual AutoML for users already familiar with sklearn -- I was able to understand the system quickly, and from the user study, it appears that other new Lale users were able to pick it up easily as well.

I am not sure whether the translation scheme from Lale syntax (combinators and schemas) to a form that can be given to an optimizer is as novel as the authors claim. Of course, given that Lale is new, the implementation details of the translation scheme can be considered new as well, but the core ideas in the translation scheme seem standard. The normalize step is quite common when dealing with conjunctions and disjunctions. The combine step is straightforward as well, simply putting the schemas into the hierarchical pipeline structure defined by the combinators. Once we have a hierarchical nesting of search spaces, flattening and discretizing are “obvious” steps to take if a particular optimizer requires a flattened and/or discretized representation. It is good that the paper describes these steps in detail, and providing such detail can count as a contribution of the paper, but I would not describe the translation process as “novel”.

**Originality: 2/5** The authors came up with a well-designed and easy to use syntax for enabling gradual AutoML in sklearn, although the ideas behind this syntax (the pipe/and/or combinators, schemas for hyperparameters) are quite straightforward (which is a good thing for usability!). It also seems that the implementation details to make their syntax work, i.e., the translation scheme, are not completely original either. Ultimately though, it appears that Lale is the first successful (easy to learn and understand, performs well, used in practice) framework for AutoML that naturally supports gradual automation (is this true?), with its usage in industry as further evidence of its success. In my opinion, accounts of such “success stories” are important research contributions, even if their technical details are not entirely original.

**Quality: 4/5** The proposed Lale system itself is technically sound. My only concern is about the empirical results, where I would like more detailed analysis of the results, what conclusions we can draw, and discussion about why those conclusions make sense. See my questions below.

**Clarity: 5/5** The paper was a pleasure to read. The ideas are organized and communicated very effectively.

**Significance: 4/5** This work is quite significant, since it enables more users to easily use gradual AutoML, which will then improve whatever work those users end up doing. I am unsure about the significance of the empirical results though, since Lale appears to perform slightly worse than the SOTA auto-sklearn. 

## Questions
* It is currently not clear whether Lale produces better or worse accuracies than auto-sklearn -- the paper simply mentions that they are in the same ballpark. It seems that Lale gets a slightly worse accuracy on a majority of tasks, but performs much better on the shuttle dataset. I’m assuming that auto-sklearn simply has a bug/flaw that manifests on the shuttle dataset, and the nature of that bug/flaw is unrelated to this paper. Ignoring the shuttle dataset, Lale appears to perform worse than auto-sklearn overall -- why would this be the case?
* Why would GridSearchCV and Halving-GridSearchCV time out in Table 3? Does Lale produce too large of search spaces for those methods? How do other approaches deal with GridSearchCV?

## Other suggestions
*  In Table 2, it would be nice to have stats aggregated over all datasets, so there is one number for comparison between the different AutoML methods. However, doing this fairly is tricky. For instance, Lale dominates in performance on the shuttle dataset, so simpy using a mean of the numbers in a column would be skewed toward that result.
* Since auto-sklearn is the primary baseline for comparison, the paper would benefit from a short description of how auto-sklearn works and the differences between it and Lale.
* Line 368: interoperates -> interoperate

limitations_and_societal_impact: The authors' Section 7 is more than adequate.
review_title: null

Meta-review: This paper elicited significant discussion. The reviewers agreed that the proposed framework could be a valuable tool and could potentially increase the uptake of AutoML systems. The user study, while small, also boosted confidence in the system's utility. The paper is also well-written. 

On the negative side, while the framework is novel in aggregate, the individual components of the framework mostly reflect known ideas. Also (less importantly in my opinion), the paper does not adequately demonstrate that this framework is competitive in performance with state-of-the-art, non-pipeline-based AutoML systems.

In the end, though, the paper's positives outweigh the negatives. After calibrating across my pile, I am recommending acceptance. Please make sure to incorporate the discussion in the review thread in the paper's final version.

Decision: Accept (Poster)