Pipeline Combinators for Gradual AutoML
Authors: Guillaume Baudart
Martin Hirzel
Kiran Kate
Parikshit Ram
Avraham Shinnar
Jason Tsay
Venue: NeurIPS 2021 Poster (2021)
Keywords: AutoML
scikit-learn
programming models
functional programming

Abstract: Automated machine learning (AutoML) can make data scientists more productive.  But if machine learning is totally automated, that leaves no room for data scientists to apply their intuition.  Hence, data scientists often prefer not total but gradual automation, where they control certain choices and AutoML explores the rest.  Unfortunately, gradual AutoML is cumbersome with state-of-the-art tools, requiring large non-compositional code changes.  More concise compositional code can be achieved with combinators, a powerful concept from functional programming.  This paper introduces a small set of orthogonal combinators for composing machine-learning operators into pipelines.  It describes a translation scheme from pipelines and associated hyperparameter schemas to search spaces for AutoML optimizers.  On that foundation, this paper presents Lale, an open-source sklearn-compatible AutoML library, and evaluates it with a user study.

Pipeline Combinators for Gradual AutoML
Guillaume Baudart
Martin Hirzel IBM Research, USA Miize@uusibmm.omm
Kiran Kate IBM Research, USA kakate@us.ibm.com
Inria, ENS - PSL University, France
httplaume.baudrr@@iimn
Parikshit Ram IBM Research, USA
Avraham Shinnar IBM Research, USA shinnar@us.ibm.cd wtt.puus..b..oommmmm
Jason Tsay IBM Research, USA
parikshit.ra@iibmccm
aaso.tsayyibb.com
Abstract
Automated machine learning (AutoML) can make data scientists more productive. But machine learning is totally automated, that leaves no room for data scientists to apply their intuition. Hence, data scientists often prefer not total but gradual automation, where they control certain choices and AutoMI. explores the rest. Un- fortunately, gradual AutoMI. is cumbersome with state-of-ther of-the-art tools, requiring large non-compositional code changes. More concise compositional code can be achieved with combinators, a powerful concept from functional programming. This paper introduces small set of orthogonal combinators for composing machine- learning operators into pipelines. describes translation scheme from pipelines and associated hyperparameter schemas to search spaces for AutoML optimizers. On that foundation, this paper presents Lale, open-source skleaan-compatbbl AutoML library, and evaluates it with a user study.
Introduction
Automated machine learning can make data scientists more productive by lowering the barrier to entry for novices as well as reducing the tedium for experts. With rotal automation, the data scientist just provides data and the AutoML tool does the rest. While this yields a convenient starting point, gives users little control. Instead, gradual automation, where the user tweaks some choices by hand, leverages user expertise for better outcomes and wastes less compute time. Unfortunately, gradual automation is not easy with existing AutoML tools and their programming interfaces. Existing AutoML tools tend to have different interfaces from those for manual machine learning, preventing gradual transition between the two. Furthermore, with existing tools, code changes for gradual automation tend to be large and non-compositional. The programming interface of existing tools often coupled given optimizer. And finally, tools often support controlling some aspects of automated pipeline search only with a much lower-level programming interface, at all.
We hypothesize that gradual AutoML first discovered by Schönfinkel in the 1920s in an attempt to find elementary building blocks of logic [41]. Combinators functions that compose functions and play important role in functional programming. For instance, Lisp provides the combinators map and reduce [43], which inspired the MapReduce programming model [11, 22). Combinators enable facit programming style, where data remains unnamed, keeping function compositions concise and minimizing edits when re-arranging them. We argue that sklearn's 's make_ tike_ppppline and make_ make_union union [7] are combinators: they compose machine-learning operators into pipelines without naming datasets such as X or y during composition. These two combinators enable the primary execution modes, training and prediction, of sklearn pipelines. This paper streamlines them with infix notation, adds choice combinator, and adds AutoML search third execution mode.
be made casier via combinators. Combinators were
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Enabling search as third execution mode requires translation from combinators to AutoML search spaces. To solve this challenge, this paper describes a novel translation scheme, which takes a combinator-basee based pipeline and generates a search space suitable for given AutoMI. optimizer. The translation first normalizes and combines the hyperparameter schemas of the operators in the pipeline, which specify the space of hyperparameter values. Next, depending on the optimizer at hand, it flattens and discretizes the search space. This translation is essential to bridge from the modular and tacit combinators syntax to optimizer's syntax, and may re-arrange components and introduce (possibly mangled) names as needed. The translation scheme also supports round-tripping back to combinators, needed for evaluating and understanding results well as for iterative refinement.
This paper introduces Lale, a Python AutoML library implementing the combinators and the transla- tion scheme. Lale enjoys active use both in the open- community and in IBM's AutoAl product. It is inrearnncompatbble giving users access to many operators and familiar functionality. Lale is gradual, letting users specify only what they want while reusing and automating the rest. For instance, Lale comes with an extensive library of reusable hyperparameter schemas for many popular operators, users rarely need to write their own schemas; but it also makes easy to customize schemas when desired. There are Lale optimizer backends for Hyperopt [6] (with 4 solvers tpe, atpe, rand, and anneal); sklearn's 's GridSearchCV and VridSernGrdSSarchCCV [7]; ADMM [26] (with 5 continuous and 5 combinatorial solvers); SMAC [15]; and Hyberband [25].
The contributions of this paper are:
Combinators for gradual automated machine learning.
A translation scheme for mapping combinators to a broad variety of optimizers.
The Lale library and a user study for validating combinators and gradual automation.
This paper addresses the problem of simplifying gradual AutoML. Our solution, Lale, easy to use for data scientists with moderate sklearn experience, as demonstrated by the user study. Lale uses combinators for modularity, and uses schemas both for search spaces and for type-checking. It is more expressive than the high-level interfaces of prior AutoML tools. Lale gives users fine- ine-grained control over AutoMI. without requiring them to be AutoMI. experts.
2 Pipeline Combinators
This paper proposes three combinators to compose machine-learning operators into pipelines. To support hyperparameter tuning, each operator has associated hyperparameter schema. Thus, when combinators compose operators, then under the hood, that implics weaving together the associated hyperparameter schemas. This section deseribes the syntax both for combinators and for schemas, leaving it to Section 4 to describe the translation that weaves them together for AutoML.
Combinators. The three combinators are >>, &, and The pipe combinator, p> >> q, connects the output of to the input of q. similar to sklearn's is make_ paaeeeeeppeine.. The and combinator, p & q. composes p and q side-by-side without creating dataflow between them. is similar to sklearn's make_union, except that make_ (p, q) implicitly concatenates the output features of and whereas p does not. Instead, if users want to combine features after & they have to explicitly add an operator for that. Making this explicit makes code more understandable and also provides flexibility to pick different data-combining operators when desired, for instance, vertical stacking vs. relational joining [40]. The or combinator, P I q. introduces a choice between p and q. While it has no direct equivalent in sklearn, it can be encoded in sklearn's 's GridSearchov, as well as in other optimizers such as Hyperopt.
Figure 1 shows the syntax for combinator-basee pipelines. The start symbol, pipeline, can be an individual operator it can nest other pipelines by using one of the three combinators. An individua/Op has a name and an optional configuration for zero or more hyperparameters. For example, in Figure 2, the operators Project and AdsBoostClaseifier are explicitly configured with hyperparameters, whereas the operators PCA, NoOp, etc. are not. When the user does not manually configure hyperparameters. then AutoML automatically tunes them. hyperParam configuration uses an expression, which can be a simple Python value or a nested pipeline. For example, in Figure 2, Project is configured with a simple Python dictionary, whereas AdaBooatClasnifie is configured with nested pipeline. An operator that takes another operator as argument is higher-order, and besides kdaBoostClassiiierr
2

---

pipeline individualOp pipe | and or
pre_n
(Project Project(colummey (columns= ("type" "number"] "}>
>> (PCA NoDp))
individualOp ::= NAME NAME (hyperParam
hyperParam == NAME expression
pre_a
(Project (columns= (aatumns==ttppe"m"""""a ("type": "tyyp""""srinng""" )
>> (DrdinalEncoder [neffotEncoder)
expression == VALUE pipeline
pplan ((ree_n & pre_s)
pipe
pipeline pipeline
ConcatFeatures
and or
pipeline pipeline
>> AdaBoostClassifier(
pipeline pipeline
base_estimator= (J48 LR)))
Figure I: Pipeline syntax. Non-terminals are in lower-case italics, literal tokens (G); (6), >>, &, !) are in type-writer font, other terminals are in all-caps ITALICS, and the meta-syntax (:=, indicates rules, choice, and Kleene star.
Figure 2: Pipeline example with two prepro- cessing sub-pipelines. Features from both are concatenated and piped to a boosted ensemble, whose base estimator is a choice of classifiers.
there are plenty of other examples, such as sklearn's RFE, OneVeResstCasesifiee, or ColumnTraneforoo or fairness mitigation operators with preprocessing operators as argument [19].
What makes the combinator-based pipeline syntax powerful is that everything nests. A pipeline is itself an operator in the sense that it implements the methods fåt and transform or predict. Unlike other AutoML tools, Lale can thus easily express search spaces around higher-order operators. The syntax is concise, as it focuses just on operators and is tacit about data. The Lale library implements this syntax as pure Python code. That has the advantage that users need not learn a new programming language, and can instead make full use of Python's 's existing features. For example, Figure 2 uses a Python assignment (pre_n ...) to improve readability. Furthermore, existing Python tooling just works, such as auto- completing editors, syntax and type checkers, and interactive notebooks.
Execution modes. Pipelines have three execution modes: predict or transform; fit; and AutoMI. search. The first two modes come from sklearn. Their semanties based the dataflow graph of a pipeline, which is directed acyclic graph whose vertices are operators and whose edges are given by the >> combinator (note that and do not induce dataflow edges, and pipelines with only support the third execution mode). The first execution mode of a pipeline, predict or transform, simply invokes predict or transform on the vertices in topological order, guaranteeing that the results from all predecessors of an operator are available before that operator fires. The second execution mode, fit, also processes vertices in topological order, calling first fit and then transform or predict. The result of fit on a pipeline is a new pipeline that is the same as the original except that all operators are replaced by trained versions, i.e., their learned coefficients are bound. Higher-order operators have the flexibility to invoke the methods of nested operators as needed for the outer operator.
The third execution mode of pipeline, AutoML search, is implemented by translating the planned pipeline to a search space for an AutoML optimizer. The search space encodes joint operator selection and hyperparameter tuning. Given a planned pipeline such as pplan from Figure 2, users access this execution mode by invoking the auto_confinn igure method:
beat_found pplan. cuto__configure train_X, train_y, optimizer Ryypero,rrtypprrot,, cv-3, max_opt__time t00eemm-0m max_eval__t t0me=30)
2
The result (in this case best_ is the trained pipeline that yielded the best score (here, cross- validated with ev-3) during the search. Section 4 discusses the details of the translation.
Schemas. Besides combinators, which specify space of pipelines to search, AutoMI. also needs hyperparameter schemas, which specify a space of hyperparameter values to search. Here, we adopt the value-set definition of a schema (or type) as a space (or set) of values [33]. While data scientists frequently write code with combinators to exercise control, they write schemas less often. As of today, Lale includes schemas for 216 operators, sufficient for many common machine learning tasks.
Figure shows the most important parts of the schema syntax (eliding some details, such as defaults, integers, and array schemas, that are not needed for examples in this paper). The start symbol, schema, can be an enumeration, a range, an object, or it can nest other schemas by using one of three logic connectives. An enum schema specifies a categorical hyperparameter with choice between one or more values. Figure includes multiple examples of enums: [mle]. trueefalsee, [0.25]. [linear, sag, lbfgs), etc. A range schema specifies continuous hyperparameter with a minimum and

---

schema : ::=mmm] range| object
disj
conj neg
PCA: N: (0..1) V [mle]}
enum := [ VALUE* ]
J48 dict [true, false,, C: (0..0.5)
range = < NUM NUM) distr
(dict( R: [true]] + dict(C: [0.25]
distr = uniffrrm" "loguniform"
LR dS [linear, sag, lbfgs] P: 111, 1
object = dict { (NAME: schema)
(dictí(: [sag, lbfgs]} - dict<P: [[2}})
disj = schema v schema
conj == schema A schema
Figure 4: Schema examples for 3 operators (sim- plified): sklearn's PCA and LogisticRogreessoon and Weka's 's J48. We write = t for (-s) Vt.
neg
= schema
Figure 3: Schema syntax.
maximum (most AutoML optimizers do best with bounded ranges) and an optional distribution distr. Figure 4 includes two examples of ranges: (0..1) and (0..0.5). An object schema specifies dictionary of zero or more hyperparameter schemas. For example, Figure 4 specifies one hyperparameter for the PCA operator from sklearn [7] and two each for J48 from Weka [18] and for LR from sklearn.
The schema syntax supports three logic connectives disjunction (V), conjunction (A), and negation (-). Disjunction describes union type: for example, in Figure 4, the N hyperparameter of PCA can specify the number of components with either a range (0..1) or an enum imle). Conjunction describes an intersection type and negation can be used to describe an implication. These connectives help express constraints for ruling out combinations of hyperparameters that are individually valid but collectively invalid. For example, with Weka's J48 operator, if R is true (use reduced error pruning), then C must be 0.25 (cannot set the pruning confidence to other values). Figure 4 specifies this via +, which is syntactic sugar for the basic logic connectives. Hyperparameter constraints are useful for pruning invalid areas from the search space, so AutoML search can focus on valid areas instead.
To make schemas easier to use, we decided not to invent our own new schema language, but instead to adopt a mature existing language with a broad user base and tooling ecosystem. One choice we considered was Python 3 types [45, 37]. Unfortunately, Python 3 types are not expressive enough for specifying constraints. But fortunately, there is another language that also works well with Python and is used extensively by large community well beyond that of AutoML: JSON Schema [35]. Most constructs from Figure 3 have exact equivalents in JSON Schema. For instance, a range (a..b) can be written as ("type": "number", "minimum" :a, "maximum" :b). Lale adds a few extensions to JSON Schema. For instance, it supports an optional "distribution" keyword a "number" " schema; and supports "operator" type for hyperparameters that accept a nested machine-learning operator.
3 Gradual Automation
The previous section described pipeline combinators and hyperparameter schemas. This section builds on that foundation to describe a programming model for gradual AutoML. Here, gradual means that users get to use total automation if they so desire, but can also exercise fine-grained manual control over any aspects of the pipeline. The programming model design was guided by three principles. First, progressive disclosure lets users start by learning just subset of the constructs, only requiring them learn more constructs as needed to accomplish more tasks. Second, orthogonality minimizes the number of independent constructs to make the programming model easy to learn, while maximizing the ways these constructs can be plugged together make the programming model expressive [27]. Third, the principle of least surprise makes each construct look and behave the way most users would expect. That why we started from well-established programming models sklearn [7] and JSON Schema [35] and try to be faithful to their expectations.
Figure show the learning curve of gradual AutoML from zero to mastery. The rest of this section describes each set of constructs, starting at the bottom and working upwards one layer at a time.
Use total automation. Lale provides an AutoPipeline operator. From the outside, it looks and feels just like any sklearn operator with fit and predict methods. Internally, fit runs AutoML on a predefined planned pipeline, which users need not dwell on. Hence, total automation with Lale is trivial to learn for users who already know skleam (least surprise).
Inspect returned pipeline. One level up from total automation is to look at its result: even users who have little interest in controlling the pipeline may want to see for transparency. Lale provides a visualize method and round-trippable pretty_ pprint method that returns pipeline's ''s Python code.

---

mastery
add new optimizers
new search space encoder ssarch point decoder
write pipeline grammans
rocursive operator definition, sample, unfeld
add operntors
me nake_ make_oppeeeeeeeeee operator (irpl, achemas)
control search space details
freese, customise oeteooneeeeeeeeeeeeeeeeeee schema
refine returned pipeline
functions that decompase pipelines
plan pipeline
1 & operalor library, auto cnte_configure
inspect returned pipeline
visualise, peattyprima petoPipeline, rit, predict
use total automation
Figure 5: Gradual automation with progressive disclosure.
Plan pipeline from scratch. Learning about the combinators and the auto auto_conf igure method from Section 2 brings users to the next level. To further assist in using Lale's collection of 216 operators, Lale generates documentation from schemas and uses schemas for error checking [17, 38].
Refine returned pipeline. Data science, even with automation, is often iterative. After an AutoMI. search (whether with full automation or with hand-planned pipeline),
may want to tweak the pipeline for another search. Lale supports this with functions for breaking down pipeline into smaller pieces, which can then be composed back into new pipeline using combinators (orthogonality).
Control search space details. To exercise even more control in a localized and composable way, users can disable AutoML search for sub-pipeline with freeze fweeeeerraieeeeeeeeee trainable, or they can disable refitting with freeze_trained eters of operator, such adjusting a range. Given returns a new operator that is the same as & except for the given schema tweaks. Again, these constructs are orthogonal combinators: frozen customized operators can be used like any others.
trained. Furthermore, users may want to locally tweak the schema for some hyperparam-
operator calling customizee act.eeaa__ceee.. customiiee__hhema...
Add new operators. Given an operator without schema, Lale infers simple schema that just sets all hyperparameters to their defaults. Users can then call customize_ _cchema to make them selectively tunable. Or they can write a comprehensive schema and attach it to the operator using _akk__opeaator
Write pipeline grammars. Some AutoML tools [31, 10, 12] explore not a rigid template for pipeline topologies but possibly unbounded space of topologies, including some that define context- free grammar using Lale's 's combinators [20, 28]. Lale lets users define such search spaces by declaring recursive operators, which are self-referential definitions (e.g., g.p g. q). Recursive operators are orthogonal to the other constructs such as combinators, with which they can nest. The methods sample e and unfold turn these into normal Lale pipelines suitable for auto_configure.
Add new optimizers. Only few users get so close to mastery that they desire to add their own optimizers. While the details will vary on case-by-case basis, the next section describes a principled translation scheme that helped Lale target several existing optimizers.
4 Translation Scheme
This paper adds a third execution mode to sklearn-style pipelines: the first two modes are prediction and training, and the third mode is AutoML search. Lale's API for this mode is auto_conf igure, search space for given optimizer and then runs that optimizer.
which first translates the pipeline
Figure details that interaction to illustrate how the translation scheme (encode and decode) interacts with an optimizer and how it uses the first two execution modes (predict and fit). The starting point planned pipeline, which has operators whose hyperparameters need to be tuned or operator choices (I combinator) whose operators need to be selected. Step "encode search space" translates the planned pipeline to an optimizer-specific search space. Once called, the optimizer loops over trials, which are search points it selects based on its solver, e.g., TPE [5]. Step "decode search point" turns an optimizer-specific search point (i.e., a specific value combination in the search space) into configuration needed for instantiating trainable pipeline. In trainable pipeline, all hyperparameters
5

---

and operator choices are bound to specific values. Fitting the trainable pipeline to training data (folds) yields trained pipeline, where the learn- able coefficients of all operators are also bound. The next step is to call predict on the trained pipeline and to score the result to compute a loss for the optimizer. After wrapping up its loop, the optimizer returns the best trial, and configure returns the corresponding trained pipeline.
aulo ronfigure
optimizer
PlannedPipaline
space
optimize
run trial
decode search point
TrainablePipeline
TrainedPippeline
IIt
create
predict
retum loss
The translation scheme is one of the contributions of this paper. Its job is to enable the modular combinator-basee syntax for users, by weaving together the operators and their hyperparame- ter schemas and rewriting them to a
return best trial
TrainedPipeline
Figure 6: Interaction with an optimizer.
form suitable for given optimizer. Users thus make local changes using orthogonal constructs without worrying about their impact on the rest of the pipeline. To target a variety of optimizers, we structured the translation scheme as series of rewriting passes between different forms. Each form has its own syntax restricting how a search space can be expressed. Figure shows the starting form. Each pass is designed to be semantics preseeriig......g......... in the sense that the search space encodes the same set of pipeline instances before and after the pass. The running example for this section is the pipeline LR); Figure 4 shows its schemas in starting form.
(J48
Normalize. Figure shows the normal form, which a disjunction of objects. This form is crafted to simplify later rewriting passes by letting them start from less feature-laden form. While nesting still possible in normal form, it is restricted to objects. Also, once in normal form, schemas contain no conjunctions or negations, which would inhibit local reasoning. Normalization works in bottom-up pass. At each level, it performs disjunction hoisting using rewrites such as De-Morgan's 's law. Negation and conjunction
pushed into primitive schemas. This temporarily give rise uninhabited or trivial nested schemas, which the pass simplifies away with another set rewrites. Our translation scheme normalizes the schema of each operator in a pipeline separately.
schema object (v object)*
PCA dict(N: (0..1)} V dict [mle]}
object == dict (NAME: prop)
J48 dict R:
false). C: (0.005)33
prop enum range object
dict R:
trus,false] C: [0.25]}
enum [ VALUE+
LR dict((S:
linear), P: (1, 12
range = C NUM NUM) dist?
dict(S:
|lieear sag, 1bfgs], P: [2]}
distr = uniforrm "loguniform"
Figure 7: Normalized schemas: syntax (left) and example (right).
Combine. Figure 8 shows the combined form, which represents the entire pipeline together instead of each operator separately. Combining encodes combinators follows. The >> and & combinators turn into an object schema (dict), where each operator becomes step 8;. The combinator turns into a disjunction schema (V), with an additional discriminant D to track what was chosen. Higher-order operators yield nested object schemas (dict). The encoding preserves sufficient information to enable later decoding. After this pass, the encoding is conceptually similar to that of Hyperopt [6], so our implementation provides a simple backend translating it into a Hyperopt search space.
Flatten. Most other optimizers require a less hierarchical search space than the combined form. Figure shows the flattened form, which is a disjunction objects without further nesting. The flatten pass hoists properties from nested objects, renaming them as needed for uniqueness (in way that enables later decoding). The flattened form is close to what SMAC [15] needs, except that objects of different disjuncts have different properties. In SMAC, all disjuncts have the same properties, with

---

so: (icc(N:(OO..))) dict [mle]}
space =
object
dict-(D: [J48] R: [false), C: (0..0.5)}
object =
dict (NAME: prop)
dict
dictCD: J48]. R: truue C: [0.25]]
prop = enum range disj
$1;
diet D: [R), S: [inear], P: [1 12 V
disj
object (v object)
dict(D: [R), St
[linear,sae sag. lbfgs). P: [12]]
Figure 8: Combined search space: syntax (left) and example (right).
dict[N: (0..1), , D: J48] R: false],
C: (0..0.5)
V dict([N: (0..1), D: J48 R: true.false
C:
dict(N: mmle] D: J48 R:
false);
C: (0..0..5)
space :- object (V object)*
dict((N: [mle], D: [J48], R: True.false
[0.25] }
object = dict { (NAME: prop)" }
dict(NN: (0..1), D: [LR], S:
[lneari
P: [1, }
prop = enum range
dict<N:: N:
(0..1), D: [LR] S:
linear, sug, 1bfgs),
P: [12]
V dict- N:
mle]. D: LR S:
linear),
P: [1, 12]
}
dict- N: mlee, D: [.R). S:
|linear, sug, lbfgs),
P: [12)
Figure 9: Flattened search space: syntax (left) and example (right).
conditionals on the side indicating which variables are relevant. The SMAC backend thus generates these conditionals, and renders the space in SMAC's parameter configuration space format [42].
The ADMM backend uses a partially flattened form. ADMM [26] supports a pipeline of operator choices. The backend thus leaves the top-level spine of pipeline intact pipeline of choices. If one of the choices contains a complex operator (e.g. a nested pipeline), flattened as for SMAC.
Discretize. The optimizers GridSearchC' and HalvingGriddSearchCV from sklearn [7] require continuous ranges be made categorical. Discretization for a hyperparameter first includes the default and then samples a user-configurable number of additional values from the range based on the prior distribution. Figure 10 shows the discretized form when generating two values per range.
dict{N; [0.50, 0.01], D: [488), R: [false], C:: [0.25, 0.01]}
diet-CN: [0.50. 0.01 D: [J48]. R: (true, C: |0.25]]
space == object (v object)* *
V dict(N:: [mle], D: [488]. R: [false), C: [0.25, 0.01]
object = dict( (NAME: enum)
V
dict N: [mle], D: [LRR, [linear, sag, 1bfgs), P: [(]]}
Figure 10: Discretized search space: syntax (left) and example (right).
Results
This section presents experiments evaluating Lale's usability and expressiveness. While the experi- ments focus on tabular data and do not use deep learning, Lale also works with other data modalities such text [8], code [29], and time-series [39], including with pipelines that contain deep-learning operators.
RQI: How usable is gradual AutoML with combinators?
To explore this research question, we compare Lale to the popular sklearn machine learning framework in user study with 18 participants. The study design is between-subjects [9], where participants are randomly assigned to a Lale or sklearn version of the study. We asked participants to perform four tasks involving sample machine learn-
Project
Normalizer
Cuncal-
K- Neighbars- Classiffer
Solect-
Project
Best
Figure 11: Sample pipeline used in user studies.
ing pipelines: (T1) inspecting and understanding an existing pipeline, (T2) refining the given pipeline by replacing operators and hyperparameters, (T3) debugging a failing pipeline, and (T4) searching across a space potential pipelines. The tasks are based on the principle of progressive disclosure in that we provide sample pipeline (Figure 11) using common manual constructs in either sklear or
7

---

Lale format and ask participants to perform tasks that use progressively expanding set of constructs. All participants self-reported having at least some sklearn experience, worked at IBM, and were not further compensated for their voluntary participation. The supplemental material describes the detailed study design and methodology, including the sklearn and Lale notebooks used.
One surprising result of the user study is that four (44%) of the sklearn participants were not able to correctly identify the branching structure of the preprocessing subpipelines of the sample pipeline during Tl (seen in Figure 11), compared to a single Lale participant (11%). Two of the Lale participants also used inspection features in Lale such as the vieualize and pretty-priws methods. In T4, two (22%) of the sklearn participants were either unable to correctly (or at all) implement some method of searching across a space of potential pipelines whereas all of the Lale participants were able to complete the task. When implementing T4, most of the Lale participants (6 or 67%)
used the or combinator (1) and the AutoMI. execution mode to perform automated search as opposed to more manual methods. This is compared to two (22%) sklearn participants using automated search via GridSearchCV. The relatively high success rate of Lale participants both in understanding existing pipelines that use the and combinator (4) and in composing new planned pipelines that use the or combinator (1) suggests that new Lale users are able to pick up and use these independent constructs. This supports the principle of orthogonality in the Lale programming model. Automated search also tends to be more concise than manual implementations for the search task, perhaps contributing to Lale participants (59% less) compared to sklearn participants. Lale participants also on average performed all of the tasks faster than sklearn participants. over 2.5 minutes faster total (13% faster). (In reality, the time difference would likely be even higher, since we stopped participants who take a manual approach early, see supplement for details.) For T2 and T3, participants performed equally well in both versions of the study. We note that some Lale participants were able to refine pipelines without referring to documentation, relying on their sklearn knowledge replace operators and define their hyperaarameters. This supports the principle of least surprise in that Lale operators work in ways that experienced sklearn users expect. Table summarizes the results from the user study.
average implementing the task in 25 fewer lines of code (LoC)
Table 1: Summary of user study results.
VERSION
COUNT
TI CORRECT
T4 CORRECT
T4 LoC MEDIAN (ALL))
TOTAL TIME MEAN (STDDEV)
Lale
9
89%
100%
10
8,9,8 9, 10, 10, 10, 75)
17:36 (5:03)
sklearm
56%
78%
23 15, 16, 22, 24, 43, 100, 100)
19:26 (6:34)
Total
18
72%
89%
14
18:54 (5:45)
Answer to RQ1: In our user study, gradual AutoML with combinators was usable enough to increase task correctness by more than 20% while reducing lines of code by more than half.
Does the translation scheme work for diverse planned pipelines? To explore this research question, we used Lale's auto_configury on several planned pipelines with Hyperopt and its TPE solver. We used auto-sklearn [15] as a state-of- art baseline; it uses SMAC for joint algorithm selection and hyperparameter tuning similar to our work, but also uses meta-learning, which we do not yet do. We used a 2.0GHz virtual machine with 32 cores and 128GB memory and gave each search 1 hour time budget with a timeout of 6 minutes per trial, which corresponds to the default setting of auto-sklearn. We chose 14 datasets from OpenML [46] (CC-BY license) that allow for meaningful optimization (as opposed to just the initial few trials) within that 1-hour budget. Our datasets are drawn from the AutoML Benchmark [16] and four of them were also used in the auto-sklearn paper [15]. (The goal of our evaluation is orthogonal to the sizes of the datasets.) We used a 66:33% % train:test split with 5-fold cross validation on the train set during optimization.
RQ2:
Table 2 summarizes the results. Column AUTOSKL shows the auto-sklearn baseline; LALE-AUTE AUTO uses Lale's 's AutoPipeline operator from Section 3; LALE-AD3M and LALE-TPOT use Lale grammars inspired by AlphaD3M [12] and TPOT [31] unfolded with a maximal depth of 3; LALE ADB uses pipeline with a choice of preprocessors and the auto-sklearn [15] with the set of classifiers limited to AdaBoost with only preprocessing. (See supplement for details.) All searches succeed and most yield performance in the same ballpark as auto-sklearn (aside from shuttle, which triggers an auto-sklearn issue that other users have also encountered). Considering all 14 datasets, LAL.E-UTO performs on average 0.7% better than AUTOSKL.. Excluding shuttle, Lale performs on average 0.3% worse than AUTOSKI. These 0.3% are smaller than the standard deviation on most datasets, and based on a t-test, they are not statistically significant.
higher-order operator; and ASKL-ADB
8

---

Table 2: Test accuracy for 14 OpenML classification tasks, using 4 Lale search spaces with Hyperopt.
Absolute accuracy mean (and stddev) over 5 runs
100 x (LALE/AUTOSK) 1)
LALE-AUTO LALE-TPOT
LALE-AD3M
ASKL-ADB
AUTO
TPOT
AD3M
australian
85.1(0.4)
86.2
85.9 (0.6)
86.8 (0.0)
(1.0
(3.1)
1.3
0.9
blood
77.9114)
(0.3
77.5 (2.5)
74.7 (0.7)
77.1 (0.7)
74.7 (0.8)
-3.3
-0.5
-4.0
-1.0
breast-cancer
73.0(0.6)
73.0 (0.0)
(1.4
(3.3)
70.9 (2.0)
72.4 (0.5)
0.0
-2.3
4,9
-2.9
99.400.1)
97.7
(0.3)
-6.7
-1.1
credii-g
76.6(1.2)
75.7 (0.0)
74.1 (0.5)
74.8 (0.4)
(1.3)
76.2
-1.1
-3.2
-2.4
-0.7
diahetes
77.0(1.3)
(1.4
(0.2)
75.0
-0.9
jungle-chess
88.1().2)
(2.4
82.9
74.1 (2.0)
(2.4
86.9
4.9
0.9
-15.8
1.5
83.8(0.3)
(3.0
83.5
80.6
(0.4
(0.3)
-0.5
-0.4
-0.2
-0.6
kr-ve-pp
99.7(0.0)
(0.)
(9.0
96.8
(0.5
(0.5(
-0.2
-2.9
-0.2
mfcat-factors
98.7(0.1)
97.1 (0.0)
97.9
(0.5
97.5 (0.4)
97.9 (0.1)
-1.6
-1.5
-1.2
-1.2
phoneme
90.3(0.4)
(9.6
(0:4)
-0.2
shutle
87.3(116))
100.0 (0.0)
(9.9
(9.9
10.00
(0.00
14.5
14.5
14.4
14.6
spectf
87.9(0.9)
87.7 (0.0)
(2.2)
(6.6
(2.6)
89.7 (2.9)
-49
sylvine
95.4(0.2)
(0.0)
94.4 (0.7)
91.3 (0.1)
95.1 (0.2)
95.1 (0.1)
-0.4
-1.1
-.3
-0.3
Answer to RQ2: The translation scheme works for pipelines with complex topologies. including those generated by grammars inspired by AlphaD3M and TPOT, well as pipelines including ensembles, producing competitive results.
RQ3: Does the translation scheme work for diverse optimizer backends? To explore this research question, Table lists test accuracies using 9 different optimizers for the OpenML phoneme dataset. The planned pipeline uses scaler, polynomial feature generator, and choice between XGBoostClassifier and LGB.Classifier. The other experimental settings are the same as above. All translations succeed, and most yield reasonable accuracy. except for the ones that timed out. The translated search spaces for GridSearchCV and HalvingGridSearchee explore within the time budget. Most people would only grid-based search for very small search spaces, but then, suffer from exploring the space less thoroughly than "proper" AutoML tools.
were correct but too large to
Table 3: Test accuracy (mean and stddev) for the phoneme dataset using multiple optimizer backends.
Hyperopt Anneal
Hyperband
ADMM BOBa
GridSearchCC
Halving-
TPE
Rand
RND
GPRND
GrdSSerccCCV
88.75 (0.2)
88.94
88.75 (0.5) 87.94 (0.2)
89.00 (0.2)
(0.2)
88.61 (0.3)
Timed
Timed
Answer to RQ3: The translation succeeds for Hyperopt (3 optimizers), Hyperband, ADMM (3 opti- mizers), GridSearchCV, and HalvingGridSearchCyy
6 Related Work
Combinators, which were discovered in the 1920s [41], have long been prominent in functional programming languages such as Lisp [43]. The advantage of their tacit style has been recognized more broadly; for instance, the Unix pipe [30] can be viewed as combinator. Combinators have also enabled some sophisticated innovations such as parser combinators [23].
Combinators and data science. MapReduce [11] is successful programming model for large- scale data processing named after two combinators [22]. Allison [2] describes Haskell combinators for converting between different machine-learning model types, but does not diseuss AutoML. Sklearn [7] provides combinators muake_pipeline and make_unioz) as well as some basic AutoML (GridSearchCv, KalvingGridSearrd iSSarchhV, and BandomizedSearchc)) but only integrates combinators with AutoML loosely, instead requiring users to perform manual name-mangling and non-compositionall repetition. Lale builds on sklearn, extending and streamlining its combinators and tightly integrating them with AutoML. Pilat et al. [36] describe a combinator language suitable as a target for automatic code generation with genetic programming, but intended be written by hand. Unlike our work. they focus on total (not gradual) automation and do not provide an or combinator (1).
Auto programming models. AutA demonstrated how to support conditional hyper- parameters multiple solvers, but the paper focused on single planned pipeline, not gradual automation [44]. While both hyperopt-sklearn [21] and auto-sklearn [15] support some gradual automation, instead of tightly integrating gradual automation with combinators, they require users to

---

drop to a bespoke lower-level API: Hyperopt for hyperopt-sklear and SMAC's parameter configura- tion space format [42] for auto-sklearn. We argue that JSON Schema [35]. being a language with a track record of wide-spread use by programmers who are not AutoMI. experts is less low-level; e.g., it is the backbone of OpenAPI [32]. We also built tool to extract schemas from sklearn docstrings [4]. In Optuna, users write imperative code that first samples values, then uses either if-statements for operator choice or passes values to constructors for hyperparameter tuning [1]. In contrast, combinators are more concise, more localized, and less surprising to sklearn users. OBOE [48] proposes an alternative meta-learning approach from that of auto-sklearn; requires an initial discrete set of models with hyperparameters choose from, and Lale could potentially be used to specify this set of models. In PyGlove, users start from a manually written neural network which they then "hyperify" by replacing a component or hyperparameter with "hyper value" [34]. Their notion of "hyper" is similar to our notion of "planned", but they do not use combinators, and they foeus only on neural architecture search. AutoGluon-Tabula focuses total automation and the paper does not discuss gradual automation [13]. In AutoGOAL, users specify a hierarchy of pre-defined "indicator" types, then the system tries any pipeline of operators that gets from the input to the output [14]. AutoGOAL's API deviates from sklearn conventions, and while it uses the syntax of Python types, it does not capture constraints, nor is it compatible with Python 3 type checkers.
7 Limitations and Societal Impacts
limitation of the user study is that it used only 18 participants for the sklearn treatment plus 9 different participants for Lale), and that all participants worked at IBM. Another potential limitation that researcher bias may be present in the design of the user study tasks. We designed the tasks to showcase and validate aspects of the Lale programming model which may potentially bias tasks in favor of Lale. We attempt to mitigate this by only selecting participants who are experienced in sklearn and by only allowing Lale newcomers to participate in the Lale version of the study.
One limitation of the translation scheme that not guaranteed to handle all JSON Schema features; in fact, we encountered and fixed some limitations early during development. Fortunately, there were no new limitations recently, and there theoretical work that could be used to tackle them if they arise [3]. A limitation of schema-base interdependence between operators. We mitigate that by providing not just hyperparameter schemas but also dataset schemas, which enable us to check whether data flowing from one operator to another in pipeline compatible [17]. A limitation of the programming model is that, while achieves a high level of sklearn compatibility, Python code that explicitly reflects over dynamic types still expose differences. We mitigate this by testing that Lale interoperates well with a broad set of sklearn features including metrics, -ross-validation, operators, etc.
constraints is that they currently do not capture
One potential societal impact is that enabling automation of searching machine learning pipelines approaches to data science tasks and raise CO2 emissions.
may encourage computationally--ee On the other hand, we hope gradual automation will encourage data scientists use their domain knowledge to more selectively and efficiently automate their machine learning work. In future work, we are planning to adopt caching techniques [24, 47] to further reduce CO2 emissions.
Conclusion
This paper presents combinators for gradual AutoML, along with novel translation scheme from combinators to existing optimizers. We have implemented our combinators in an open-source library called Lale, while ensuring they interoperate smoothly with the sklearn ecosystem. Besides being open-source, Lale is also actively used in a commercial AutoMI, product, highlighting the real-world value of the underlying concepts. Our ultimate goal is to make AutoML accessible manual ML. by offering a gradual combination of both.
10

---

References
[1] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: next-generation hyperparameter optimization framework. In Conference on Knowledge Discovery and Data Mining (KDD) Applied Data Science Track, pages 2623-2631, 2019.
[2] Lloyd Allison. Models for machine learning and data mining in functional programming. Journal on Functional Programming (JFP), 15(1):15-32, January 2005.
[3] Nohamed-Amine Baazizi, Dario Colazzo, Giorgio Ghelli, Carlo Sartiani, and Stefanie Scherzinger. Not elimination and witness generation for JSON Schema. In Conférence sur la Gestion de Données (BDA), October 2020.
[4] Guillaume Baudart, Peter Kirchner, Martin Hirzel, and Kiran Kate. Mining documentation extract hyperparameter schemas. In ICML Workshop on Automated Machine Learning (AutoML@ICM)) 2020.
[5] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. In Conference on Neural Information Processing Systems (NIPS),
pages 25544,554, 2011.
[6] James Bergstra, Brent Komer, Chris Eliasmith, Dan Yamins, and David D. Cox. Hyperopt: Python library for model selection and hyperparameter optimization. Compulational Science & Discovery, 8(1), 2015.
[7] Lars Buitinek, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake VanderPlas, Arnaud Joly, Brian Holt, and Gaël Varoquaux. API design for machine learning software:
Experiences from the scikit-learn project, 2013. https://arxiv.orr/abbi
309.0238.
[8] Arunima Chaudhary, Alayt Issak, Kiran Kate, Yannis Katsis, Abel N. Valente, Dakuo Wang, Alexandre V. Evfimievski, Sairam Gurajada, Ban Kawas, A. Cristiano I. Malossi, Lucian Popa. Tejaswini Pedapati, Horst Samulowitz, Martin Wistuba, and Yunyao Li. AutoTeet: An end- to-end AutoAl framework for text. In Demonstration at Conference Artificial Intelligence (AAAI-Demos), pages 16001-16003, 2021.
[9] John W. Creswell. Research design: Qualitative, quantitative, and mixed methods approaches. SAGE publications, 4th edition, 2013.
[10] Alex G. C. de Sá, Walter José G. S. Pinto, Luiz Otavio V. B. Oliveira, and Gisele L. Pappa. RECIPE: A grammar-based framework for automatically evolving classification pipelines, In European Conference on Genetic Programming (EuroGP), pages 246-261, 2017.
[11] Jeffrey Dean and Sanjay Ghemawat. MapReduce: Simplified data processing on large clusters. In Symposium on Operating Systems Design and Implementation (OSDI), pages 137-150, 2004.
[12] Iddo Drori, Yamuna Krishnamurthy, Raoni Lourenco, Remi Rampin, Kyunghyun Cho, Claudio Silva, and Juliana Freire. Automatic machine learning by pipeline synthesis using model-based reinforcement learning and grammar. In ICML Workshop on Automatic Machine Learning (AutoML (NtoML@ICML).. 2019.
[13] Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhangh, Pedro Larroy, Mu Li, and Alexander Smola. AutoGluon- Tabularr Robust and accurate AutoML for structured data. In ICML Workshop on Automated Machine Learning (AutoML @ ICML), 2020.
[14] Suilan Estevez-Velarde, Alejandro Piad-Morfiss, Yoan Gutiérrez, Andrés Montoyo, Rafael Munoz, and Yudivian Aimeida-Cruz, Solving heterogeneous automl problems with AutoGOAL. In ICML. Workshop on Automated Machine Learning (AutoMI. @ ICMI.), 2020.
[15] Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg. Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. Conference on Neural Information Processing Systems (NIPS), pages 2962-2970 2015.

---

[16] Pieter Gijsbers, Erin LeDell, Janek Thomas, Sebastien Poirier, Bernd Bischl, and Joaquin Vanschoren. An open source AutoML benchmark. In ICML Workshop on Automatic Machine Learning (AutoMI.C @ ICML), 2019.
[17] Andrew Habib, Avraham Shinnar, Martin Hirzel, and Michael Pradel. Finding data compatibility bugs with JSON subschema checking. In International Symposium on Software Testing and Analysis (ISSTA), pages 620-632, 2021.
[18] Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. The WEKA data mining software: An update. SIGKDD Explorations Newsletter, 11(1):10-18, November 2009.
[19] Martin Hirzel, Kiran Kate, and Parikshit Ram. Engineering fair machine learning pipelines. In ICLR Workshop on Responsible Al TRAI@ICLR)) 2021.
[20] Michael Katz, Parikshit Ram, Shirin Sohrabi, and Octavian Udrea. Exploring context-free languages via planning: The case for automating machine learning. In International Conference on Automated Planning and Scheduling (ICAPS), pages 403-411, 2020.
[21] Brent Komer, James Bergstra, and Chris Eliasmith. Hyperopt-skleam: Automatic hyperpa- rameter configuration for scikit-learn. In Python in Science Conference (SciPy), pages 32-37, 2014.
[22] Ralf Lämmel. Google's ' MapReduce programming model revisited. Science of Computer Programming, 70(1):1-30 2007.
[23] Daan Leijen and Erik Meijer. Parsec: Direct style monadic parser combinators for the real world. Technical Report UU-CS-2001-27, Universiteit Utrecht, 2002.
[24] Liam Li, Evan Sparks, Kevin Jamieson, and Ameet Talwalkar. Exploiting reuse in pipeline- aware hyperparameter tuning, 2019. https//aarix.orr/ab///99......
[25] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hy- perband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research (JMLR), I 18(155):1-5,, 2018.
[26] Sijia Liu, Parikshit Ram, Deepak Vijaykeerthy, Djallel Bouneffouf, Gregory Bramble, Horst Samulowitz, Dakuo Wang, Andrew Conn, and Alexander G. Gray. An ADMM based framework for AutoML pipeline configuration. In Conference on Artificial Intelligence (AAAI), pages 4892-4899, 2020.
[27] B.J. Mailloux J.E.L. Peck, and C.H.A. Koster. Report on the Algorithmic Language ALGOL 68. Springer, 1969.
[28) Radu Marinescu, Akihiro Kishimoto, Parikshit Ram, Ambrish Rawat, Martin Wistuba, Paulito P. Palmes, and Adi Botea. Searching for machine learning pipelines using context-free grammar. In Conference on Artificial Intelligence (AAAI), pages 8902-8911, 2921,
[29] Georgios Mavroudeas, Guillaume Baudart, Alan Cha, Martin Hirzel, Jim A. Laredo, Malik Magdon-Ismaill Louis Mandel, and Erik Wittern. Learning GraphQL query cost. In Conference on Automated Software Engineering (ASE) Industry Track, 2021.
[30] M. Douglas Mcllroy. A research UNIX reader: Annotated excerpts from the programmer's manual, 1971-1986. Technical report, Bell Labs, 1987. https://www..s..armmoth..cmm reader.pdf.
[31] Randal S. Olson, Ryan J. Urbanowicz, Peter C. Andrews, Nicole A. Lavender, La Creis Kidd, and Jason H. Moore. Automating biomedical data science through tree-based pipeline optimization. In European Conference on the Applications of Evolutionary Computation (EvoApplicatioons)) pages 123-137, 2016.
[32] OpenAPI Initiative, OpenAPI specification (fka swagger restful api documentation specifica- tion). 2014. http://///uhcnnnoooooooooootsttt
12

---

[33] David L. Parnas, John E. Shore, and David Weiss. Abstract types defined as classes of variables. In Conference on Data: Abstraction, Definition and Structure, pages 149-154, 1976.
[34] Daiyi Peng, Xuanyi Dong, Esteban Real, Mingxing Tan, Yifeng Lu, Gabriel Bender, Hanxiao Liu, Adam Kraft, Chen Liang, and Quoe Le. PyGlove: Symbolic programming for automated machine learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[35] Felipe Pezoa, Juan L. Reutter, Fernando Suarez, Martín Ugarte, and Domagoj Vrgoë. Foun- dations of JSON schema. In International Conference on World Wide Web (WWW), pages 263-273, 2016.
[36] Martin Pilat, Tomas Kren, and Roman Neruda. Asynchronous evolution of data mining workflow schemes by strongly typed genetic programming. In International Conference on Tools with Artificial Intelligence (ICTAI), pages 577-584, 2016.
[37] Ingkarat Rak-ammouykit, Daniel McCrevan, Ana Milanova, Martin Hirzel, and Julian Dolby. Python types in the wild: tale of two type systems. In Dynamic Languages Symposium (DLS), pages 57-70, 2020.
[38] Ingkarat Rak- aakkuunnuy,i,, Ana Milanova, Guillaume Baudart, Martin Hirzel, and Julian Dolby. Extracting hyperparameter constraints from code. In ICLR Workshop on Security and Safety in Machine Learning Systems (SecML ICLR), 2021.
[39] Subhrajit Roy, Kiran Kate, and Martin Hirzel. A semi-supervised deep learning algorithm for abnormal EEG identification. In Machine Learning for Health Workshop (MLAH @ NeurlPS), 2019.
[40] Chirag Sahni, Kiran Kate, Avraham Shinnar, Hoang Thanh Lam, and Martin Hirzel. RASL: Relational algebra in scikit-learn pipelines. In Workshop on Databases and A/ (DBAI@ NeurlPS), 2021.
[41] Moses Schönfinkel. Über die Bausteine der mathematischen Logik. Mathematische Annalen, 92(3):305-33 316, 1924.
[42] SMAC Parameter Configuration Space (PCS). http://aataml.gghhaaioossA/C/Cst mmmllpprameter-cofffgurtaan spacon--sppppsssssssees pcs.
[43] Guy L. Steele, Jr. Common Lisp: The Language. Digital Press, 2nd edition, 1990.
[44] Chris Thornton, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Browe Brown. Combined selection and hyperparameter optimization of classification algorithms. In Conference on Knowledge Discovery and Data Mining (KDD), pages 847-855, 2013.
[45] Guido van Rossum, Jukka Lehtosalo, and Lukasz Langa. PEP 484 Type Hints, 2014. https: www.ppytton..rrgdee/pppppppppppp 0484/.
[46] Joaquin Vanschoren, Jan N. van Rijn, Bemd Bischl, and Luis Torgo. OpenML: Networked science in machine learning. SIGKDD Explorations Newsletter. 15(2):49 60, June 2014.
[47] Doris Xin, Litian Ma, Jialin Liu, Stephen Macke, Shuchen Song, and Aditya Parameswaran. Helix: Accelerating human-in-the intten-----te-oooop machine learning. In Conference on Very Large Data Bases (VLDB), pages 1958-1961, 2018.
[48] Chengrun Yang, Yuji Akimoto, Dae Won Kim, and Madeleine Udell. OBOE: Collaborative filtering for AutoMI. initialization. In Conference on Knowledge Discovery and Data Mining (KDD), pages 1173-1183, 2019.
13

---

Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 7.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 7.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See supplemen- tal material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5.
(c) Did you report error bars (e.g., with respect to random seed after running experi- ments multiple times)? [Yes] See Section 5, which reports standard deviations after running experiments times.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplemental material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] See Section 5.
(b) Did you mention the license of the assets? [Yes] See Section 5.
(c) Did you include any new assets either in the supplemental material
URL?
[Yes] The Lale library is open-source at https://github. httpt/u..ooomb///le////// In addition, the supplemental material includes a copy of the code, based on snapshot from the time when we ran the experiments for this paper.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A] We used open datasets and open-source software.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] We used standard datasets from OpenMI. [46] that contained neither personally identifiable information nor offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] See supplemental material.
(b) Did you deseribe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A] There were no risks to participants.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] Participants were volunteers and did not get compensated for joining our study.
14


Reviews:
[{"review_id": "ogiUYIbxm2k", "reviewer": "Reviewer_D3TC", "summary": "This paper proposes a combinators that can combine two functions without naming datasets. They implement these combinators as a part of sklearn pipeline---both prediction and training. Finally, it also adds a hyperparameter search using AutoML to optimize the models. ", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n\n+ AutoML is an important application, and if successful, can greatly reduce data scientists' effort.\n+ It looks like a significant engineering effort and mature tool. They support many backend for the optimizers. \n+ Besides reporting testing accuracies for 14 OpenML classification tasks, they also reported  results of an user study demonstrating usefulness of the tool. \n\nWeakness\n- The high-level scientific contribution is missing. The paper provides too many low level details  without describing what are the main challenges to implement combinators.\n- All the datasets used in this paper seems rather small, and it is not clear to whether the underlying framework can support diverse model architectures.\n- It is not clear why the proposed method is contributing to AutoML. It seems like it is just combining existing AutoML approaches with the combinators. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "n7ivTK7ITDY", "reviewer": "Reviewer_cjD3", "summary": "The authors introduced a wrapper for hyperparameter optimization, named Lale, to create and verify search spaces more easily. To this end, instructions with combinators are used, which further are translated to fit the underlying optimizer (like SMAC, Hyperopt, etc.). Lale is meant to be an interface between machine and human s.t. parts of the pipeline can be easily changed, if desired.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Novelty\nThe novelty part can be narrowed down to the unified translation schema, which is the first work in the domain of AutoML to my knowledge. However, it is not clear to me whether this is more novelty in the sense of engineering some useful software or whether this sufficient scientific novelty in the method directly.\n\n# Significance / Impact\nThe advantages for Lale over existing AutoML frameworks are especially advantageous for inexperienced users due to two reasons. (i) The unified interface makes it easy to quickly set up an AutoML pipeline for any implemented optimizer. (ii) Graph overviews are a helpful and easy way to check whether the search space is defined correctly. \n\nHowever, experts in ML and AutoML tend to work as closely to the optimizer interfaces as possible to have full control over the optimizer. Since defining search spaces is transparent and easily understandable in several frameworks too, e.g. SMAC and Hyperopt, Lale would not give many advantages over these. Full AutoML systems such as AutoSklearn, on the other hand, would definitely benefit from Lale. Overall, I’m not fully convinced that there will be many users of Lale at the end of the day.\n\nFurthermore, I don’t see a clear path for scientific future work building on top of Lale.\n\nOverall, it is uncertain whether Lale will have a great impact in the community. \n\n# Soundness (method and experimental setup)\n\nThe authors described their method and experiments detailedly and the concept is sound. \n\nI appreciate the overall idea of having several API levels which allow to gradually build up expertise in using AutoML tools. Depending also on the level of the user’s expertise, this should allow to find an appropriate API without switching between different packages. However, I have the impression that conceptually this is a very high-level idea of software engineering and unclear how this scientifically contributes to the NeurIPS community.\n\n# Scholarship\nRelated work covers both combinators in general and in the AutoML domain. The domain is sufficiently covered.\n\n\n# Clarity\nThe authors introduced their AutoML library in a sorted, logical way.\n\n# Minor Comments\n* Using Lale still requires a decent background on machine learning. Calling methods and Interpreting graphs (Figure 11) may not be straight forward for inexperienced users.\n* As already mentioned in the author’s limitations, the user studies only include 9 participants, making statements not very meaningful. Additionally, the survey might even be  biased towards Lale.\n* Figure 6 shows a default routine with the only difference of “encode search space”. There’s no novelty aspect at all.\n* Table 1 would not convince me to use Lale over Auto-Sklearn.\n\n# Questions for Rebuttal\n* Table 2: It seems like Auto-Sklearn outperforms Lale-Auto. What are the reasons for that?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "GNuZexsVC3", "reviewer": "Reviewer_G6BM", "summary": "This paper aims at constructing a system for gradual AutoML that is concise, modular (or compositional), and easy-to-use. To this end, the paper introduces three orthogonal combinators (i.e., higher-order functions) which enable compositional code for gradual AutoML, and hyperparameter schemas which describe search spaces of hyperparameters. To support various backend AutoML optimizers, the paper proposes a translation scheme which translates pipelines, described by combinators and hyperparameter schemas, into search spaces for those optimizers. The paper implements these ideas into a Python library (called Lale) which includes a new execution mode (called AutoML search) for running AutoML searches. Through user studies and experiments, the paper shows that Lale is easy-to-use, and can express various pipelines and support various optimizers.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I really enjoyed reading this paper! The paper is extremely well-written and well-organized---it was easy to understand both low-level details and high-level ideas clearly. In particular, the paper succeeds at abstracting out unimportant details and explaining core ideas succinctly. Moreover, the technical contents of the paper look novel and sound to me.\n\nThe paper considers a very important problem and provides a very satisfactory solution both in technical and practical point of view. Technically, the paper introduces three combinators important for compositional gradual AutoML, among which the or combinator is particularly novel. Also the entire system (including combinators, hyperparameter schema, and translation scheme) seems to be designed and engineered in a principled way, and the paper describes some important design principles. Practically, learning and using Lale look easy to me (which is confirmed by user studies) in that it is implemented in Python and based on popular frameworks (sklearn and JSON Schema). Also Lale supports various pipeline structures and optimizer backends. For these reasons, I think Lale will have a huge positive impact on the NeurIPS community.\n\nQuestions & Comments:\n- Line 357 describes “dataset schemas” but I think it is not explained elsewhere in the paper. What is it in detail?\n- Line 291. “participants where able” --> “participants were able”.\n\n-----\n**Updates after the author response.** Thank the authors for answering my questions. I am still fond of the paper and will keep the same score. \n\n----\n**Updates after internal discussions.** I like the paper for two reasons (which I wrote in the internal discussion):\n\n* First, I do think the paper has scientific contributions. Most importantly, the paper identifies three combinators (pipe, and, or) for the domain of AutoML, and shows that they are enough to express most tasks in AutoML and further enable \"gradual\" AutoML by making a programming system for AutoML \"modular\"---note that the combinators are the key to the modularity. I think this is an important scientific contribution, and would be useful for the NeurIPS community as it enables gradual AutoML (which has not been possible in existing AutoML systems).\n* Second, I have made the following assumption: engineering software that is useful to the NeurIPS community is considered as one of the contributions NeurIPS appreciates. In this context, the paper makes an another contribution by providing the Lale library which I think would be quite useful at least for beginners of AutoML. Of course, my above assumption could be wrong and I totally understand if others disagree with it.\n\nHowever, during internal discussions, other reviewers expressed different thoughts and raised concerns on each of the two points. Given this, I suggest the authors address those concerns in the next version of the paper to make the paper stronger, e.g., by highlighting the first point more and providing concrete/quantitative evidence on how much useful the Lale library is (and will be) for the NeurIPS community.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impact are discussed in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "5ahnEpsJ71", "reviewer": "Reviewer_H7JW", "summary": "This paper propose a new autoML system with the formalization of pipeline combinators. At the high-level, machine learning can be automatically used by novices without any expertise; at the low-level, a formalization is proposed to facilitate the search of optimal hyper parameters. User studies and experiments are included to justify the designs.  ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "AutoML is a useful tool to help novices apply machine learning in their domains, thus, to design easy-to-use and efficient autoML system is an interesting research topic. This paper proposes a system with some new formalizations under its software layout. The motivation is clear: to make the utilization of the system as easy as possible while enabling efficient optimization of the hyper-parameters demanded by different machine learning pipelines.\n\nTechnically, I have a few comments:\n+ I am confused by the design philosophy  of the formalization of combinator in Section 2.  Essentially, why do we need these formalizations? The author keeps mentioning that there are some roughly equivalent operators within the sklearn toolkit. Does this section just offer some more formal definition of a language, or is the syntax defined in this section more expressive than the existing systems? It would be better to have a more explicit description about the motivation of the design.     \n+ W.r.t the grammar of \"pipeline\", I am wondering if this could make the formalization be able to be used for neural network architecture search? It seems there is no such discussion.\n\nThe presentation of the empirical study should be organized better: \n+ For RQ1, even though there is limited number of participants (which is fine due to the difficulty of organizing the user study), there still should be some formal description about the null hypothesis in this user study, then there should be some statistical analysis about the statistical significance of the result, e.g., one can consider bootstrap based methods to report the p-value w.r.t the hypothesis. Further, this section should be self-contained, there should be some brief introduction about the tasks.\n\n+ For RQ2 and RQ3, it seems that only the end-to-end performance boost is reported, on the other hand, it is important to understand where the performance gain comes from (perhaps based on some micro-benchmarks). In other words, the section should answer the question why the formalization of the combinators is effective for the black-box optimizer.\n\nPost rebuttal updates:\n\nI really appreciate the great effort the author has made to address my concerns. On the other hand, I hope during this procedure, the author would also feel that both my comments and comments from other reviewers could help to make it a better paper. Here are some follow-up suggestions:\n\n+ To be more clear about the scientific contribution from the very beginning of the paper, e.g., a more expressive formalization of the AutoML system. \n\n+ To organize the presentation of the empirical study following the general scientific principles; and to recruit more participants if possible so that one could draw some statistically significant conclusions.\n\n+ To provide some analysis about the performance gain and to discuss how the gain relates to the proposed design---at the end of the day, as scientists, we not only want to know if something works, we want to know why it works as well.\n \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "Not applicable.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall there is not enough support from reviewers for me to recommend acceptance.

Reviewers agreed on some real strengths in the paper, including (1) that is well-written and well-organized, and (2) that it tackles an important problem.

On #1:
* Reviewer G6BM: "I really enjoyed reading this paper! The paper is extremely well-written and well-organized"
* Reviewer cjD3: "The authors introduced their AutoML library in a sorted, logical way"

On #2:
* Reviewer G6BM: "The paper considers a very important problem and provides a very satisfactory solution both in technical and practical point of view"
* Reviewer D3TC: "AutoML is an important application, and if successful, can greatly reduce data scientists' effort"
* Reviewer H7JW: "AutoML is a useful tool [...] and efficient autoML system is an interesting research topic"

But the weight of opinion was that the paper (1) doesn't present a clear and significant scientific contribution, (2) shows limited empirical validation, and (3) doesn't concern a software package with large enough practical impact on the NeurIPS community.

On #1:
* Reviewer cjD3: "Furthermore, I don’t see a clear path for scientific future work building on top of Lale" and "I'm not convinced that the new operators can express substantially more than the formats of already existing AutoML tools [...] If the expressiveness of the format would be larger than prior work, I would have expected that we need also specialized optimizers for this [...] I have the impression that Lale could be a convenient package for new AutoML users, but this alone does not justify a NeurIPS paper"
* Reviewer H7JW: "be more clear about the scientific contribution from the very beginning of the paper, e.g., a more expressive formalization of the AutoML system."
* Reviewer D3TC: "The high-level scientific contribution is missing. The paper provides too many low level details without describing what are the main challenges to implement combinators." and "It is not clear why the proposed method is contributing to AutoML"

On #2:
* Reviewer cjD3: "the user studies only include 9 participants, making statements not very meaningful. Additionally, the survey might even be biased towards Lale"
* Reviewer H7JW: "organize the presentation of the empirical study following the general scientific principles; and to recruit more participants if possible so that one could draw some statistically significant conclusions."

On #3:
* Reviewer cjD3: "Overall, I’m not fully convinced that there will be many users of Lale at the end of the day." and "Lale is not even close to the level of pytorch. It has 225 stars on github and was forked 52 times. If that would be the level of impact we expect from a NeurIPS software paper, we will have thousands of papers of those each year."

Reviewer G6BM disagreed on #1 and #3, and felt the paper does clear the bar, but offered this follow-up commentary: "I suggest the authors address those concerns in the next version of the paper to make the paper stronger, e.g., by highlighting the first point more and providing concrete/quantitative evidence on how much useful the Lale library is (and will be) for the NeurIPS community."


The other reviewers offered several suggestions for the authors to improve the paper, in addition to those quoted above:
* Reviewer cjD3: "I liked Section 3 regarding the gradual automation which is a real problem for new (Auto)ML practitioners and I believe that there is a lot of untouched potential here. Unfortunately, this is not really the main focus of the paper."
* Reviewer cjD3: "Maybe JMLR MLOSS would be a better fit for this paper"


Overall, we weren't able to justify acceptance based either on the significance of the scientific contribution or on the practical impact of the software described here within the NeurIPS community. For that reason I recommend rejecting the paper, but I hope that the reviewers' feedback is helpful to the authors in improving it.
Decision: Accept (Poster)