Pipeline Combinators for Gradual AutoML
Authors: Guillaume Baudart
Martin Hirzel
Kiran Kate
Parikshit Ram
Avraham Shinnar
Jason Tsay
Venue: NeurIPS 2021 Poster (2021)
Keywords: AutoML
scikit-learn
programming models
functional programming

Abstract: Automated machine learning (AutoML) can make data scientists more productive.  But if machine learning is totally automated, that leaves no room for data scientists to apply their intuition.  Hence, data scientists often prefer not total but gradual automation, where they control certain choices and AutoML explores the rest.  Unfortunately, gradual AutoML is cumbersome with state-of-the-art tools, requiring large non-compositional code changes.  More concise compositional code can be achieved with combinators, a powerful concept from functional programming.  This paper introduces a small set of orthogonal combinators for composing machine-learning operators into pipelines.  It describes a translation scheme from pipelines and associated hyperparameter schemas to search spaces for AutoML optimizers.  On that foundation, this paper presents Lale, an open-source sklearn-compatible AutoML library, and evaluates it with a user study.

Pipeline Combinators for Gradual AutoML
Guillaume Baudart
Martin Hirzel IBM Research, USA Miize@uusibmm.omm
Kiran Kate IBM Research, USA kakate@us.ibm.com
Inria, ENS - PSL University, France
httplaume.baudrr@@iimn
Parikshit Ram IBM Research, USA
Avraham Shinnar IBM Research, USA shinnar@us.ibm.cd wtt.puus..b..oommmmm
Jason Tsay IBM Research, USA
parikshit.ra@iibmccm
aaso.tsayyibb.com
Abstract
Automated machine learning (AutoML) can make data scientists more productive. But machine learning is totally automated, that leaves no room for data scientists to apply their intuition. Hence, data scientists often prefer not total but gradual automation, where they control certain choices and AutoMI. explores the rest. Un- fortunately, gradual AutoMI. is cumbersome with state-of-ther of-the-art tools, requiring large non-compositional code changes. More concise compositional code can be achieved with combinators, a powerful concept from functional programming. This paper introduces small set of orthogonal combinators for composing machine- learning operators into pipelines. describes translation scheme from pipelines and associated hyperparameter schemas to search spaces for AutoML optimizers. On that foundation, this paper presents Lale, open-source skleaan-compatbbl AutoML library, and evaluates it with a user study.
Introduction
Automated machine learning can make data scientists more productive by lowering the barrier to entry for novices as well as reducing the tedium for experts. With rotal automation, the data scientist just provides data and the AutoML tool does the rest. While this yields a convenient starting point, gives users little control. Instead, gradual automation, where the user tweaks some choices by hand, leverages user expertise for better outcomes and wastes less compute time. Unfortunately, gradual automation is not easy with existing AutoML tools and their programming interfaces. Existing AutoML tools tend to have different interfaces from those for manual machine learning, preventing gradual transition between the two. Furthermore, with existing tools, code changes for gradual automation tend to be large and non-compositional. The programming interface of existing tools often coupled given optimizer. And finally, tools often support controlling some aspects of automated pipeline search only with a much lower-level programming interface, at all.
We hypothesize that gradual AutoML first discovered by Schönfinkel in the 1920s in an attempt to find elementary building blocks of logic [41]. Combinators functions that compose functions and play important role in functional programming. For instance, Lisp provides the combinators map and reduce [43], which inspired the MapReduce programming model [11, 22). Combinators enable facit programming style, where data remains unnamed, keeping function compositions concise and minimizing edits when re-arranging them. We argue that sklearn's 's make_ tike_ppppline and make_ make_union union [7] are combinators: they compose machine-learning operators into pipelines without naming datasets such as X or y during composition. These two combinators enable the primary execution modes, training and prediction, of sklearn pipelines. This paper streamlines them with infix notation, adds choice combinator, and adds AutoML search third execution mode.
be made casier via combinators. Combinators were
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Enabling search as third execution mode requires translation from combinators to AutoML search spaces. To solve this challenge, this paper describes a novel translation scheme, which takes a combinator-basee based pipeline and generates a search space suitable for given AutoMI. optimizer. The translation first normalizes and combines the hyperparameter schemas of the operators in the pipeline, which specify the space of hyperparameter values. Next, depending on the optimizer at hand, it flattens and discretizes the search space. This translation is essential to bridge from the modular and tacit combinators syntax to optimizer's syntax, and may re-arrange components and introduce (possibly mangled) names as needed. The translation scheme also supports round-tripping back to combinators, needed for evaluating and understanding results well as for iterative refinement.
This paper introduces Lale, a Python AutoML library implementing the combinators and the transla- tion scheme. Lale enjoys active use both in the open- community and in IBM's AutoAl product. It is inrearnncompatbble giving users access to many operators and familiar functionality. Lale is gradual, letting users specify only what they want while reusing and automating the rest. For instance, Lale comes with an extensive library of reusable hyperparameter schemas for many popular operators, users rarely need to write their own schemas; but it also makes easy to customize schemas when desired. There are Lale optimizer backends for Hyperopt [6] (with 4 solvers tpe, atpe, rand, and anneal); sklearn's 's GridSearchCV and VridSernGrdSSarchCCV [7]; ADMM [26] (with 5 continuous and 5 combinatorial solvers); SMAC [15]; and Hyberband [25].
The contributions of this paper are:
Combinators for gradual automated machine learning.
A translation scheme for mapping combinators to a broad variety of optimizers.
The Lale library and a user study for validating combinators and gradual automation.
This paper addresses the problem of simplifying gradual AutoML. Our solution, Lale, easy to use for data scientists with moderate sklearn experience, as demonstrated by the user study. Lale uses combinators for modularity, and uses schemas both for search spaces and for type-checking. It is more expressive than the high-level interfaces of prior AutoML tools. Lale gives users fine- ine-grained control over AutoMI. without requiring them to be AutoMI. experts.
2 Pipeline Combinators
This paper proposes three combinators to compose machine-learning operators into pipelines. To support hyperparameter tuning, each operator has associated hyperparameter schema. Thus, when combinators compose operators, then under the hood, that implics weaving together the associated hyperparameter schemas. This section deseribes the syntax both for combinators and for schemas, leaving it to Section 4 to describe the translation that weaves them together for AutoML.
Combinators. The three combinators are >>, &, and The pipe combinator, p> >> q, connects the output of to the input of q. similar to sklearn's is make_ paaeeeeeppeine.. The and combinator, p & q. composes p and q side-by-side without creating dataflow between them. is similar to sklearn's make_union, except that make_ (p, q) implicitly concatenates the output features of and whereas p does not. Instead, if users want to combine features after & they have to explicitly add an operator for that. Making this explicit makes code more understandable and also provides flexibility to pick different data-combining operators when desired, for instance, vertical stacking vs. relational joining [40]. The or combinator, P I q. introduces a choice between p and q. While it has no direct equivalent in sklearn, it can be encoded in sklearn's 's GridSearchov, as well as in other optimizers such as Hyperopt.
Figure 1 shows the syntax for combinator-basee pipelines. The start symbol, pipeline, can be an individual operator it can nest other pipelines by using one of the three combinators. An individua/Op has a name and an optional configuration for zero or more hyperparameters. For example, in Figure 2, the operators Project and AdsBoostClaseifier are explicitly configured with hyperparameters, whereas the operators PCA, NoOp, etc. are not. When the user does not manually configure hyperparameters. then AutoML automatically tunes them. hyperParam configuration uses an expression, which can be a simple Python value or a nested pipeline. For example, in Figure 2, Project is configured with a simple Python dictionary, whereas AdaBooatClasniffe is configured with nested pipeline. An operator that takes another operator as argument is higher-order, and besides kdaBoostClassiiierr
2

---

pipeline individualOp pipe | and or
pre_n
(Project Project(colummey (columns= ("type" "number"] "}>
>> (PCA NoDp))
individualOp ::= NAME NAME (hyperParam
hyperParam == NAME expression
pre_a
(Project (columns= (aatumns==ttppe"m"""""a ("type": "tyyp""""srinng""" )
>> (DrdinalEncoder [neffotEncoder)
expression == VALUE pipeline
pplan ((ree_n & pre_s)
pipe
pipeline pipeline
ConcatFeatures
and or
pipeline pipeline
>> AdaBoostClassifier(
pipeline pipeline
base_estimator= (J48 LR)))
Figure I: Pipeline syntax. Non-terminals are in lower-case italics, literal tokens (G); (6), >>, &, !) are in type-writer font, other terminals are in all-caps ITALICS, and the meta-syntax (:=, indicates rules, choice, and Kleene star.
Figure 2: Pipeline example with two prepro- cessing sub-pipelines. Features from both are concatenated and piped to a boosted ensemble, whose base estimator is a choice of classifiers.
there are plenty of other examples, such as sklearn's RFE, OneVeResstCasesifiee, or ColumnTraneforoo or fairness mitigation operators with preprocessing operators as argument [19].
What makes the combinator-based pipeline syntax powerful is that everything nests. A pipeline is itself an operator in the sense that it implements the methods fåt and transform or predict. Unlike other AutoML tools, Lale can thus easily express search spaces around higher-order operators. The syntax is concise, as it focuses just on operators and is tacit about data. The Lale library implements this syntax as pure Python code. That has the advantage that users need not learn a new programming language, and can instead make full use of Python's 's existing features. For example, Figure 2 uses a Python assignment (pre_n ...) to improve readability. Furthermore, existing Python tooling just works, such as auto- completing editors, syntax and type checkers, and interactive notebooks.
Execution modes. Pipelines have three execution modes: predict or transform; fit; and AutoMI. search. The first two modes come from sklearn. Their semanties based the dataflow graph of a pipeline, which is directed acyclic graph whose vertices are operators and whose edges are given by the >> combinator (note that and do not induce dataflow edges, and pipelines with only support the third execution mode). The first execution mode of a pipeline, predict or transform, simply invokes predict or transform on the vertices in topological order, guaranteeing that the results from all predecessors of an operator are available before that operator fires. The second execution mode, fit, also processes vertices in topological order, calling first fit and then transform or predict. The result of fit on a pipeline is a new pipeline that is the same as the original except that all operators are replaced by trained versions, i.e., their learned coefficients are bound. Higher-order operators have the flexibility to invoke the methods of nested operators as needed for the outer operator.
The third execution mode of pipeline, AutoML search, is implemented by translating the planned pipeline to a search space for an AutoML optimizer. The search space encodes joint operator selection and hyperparameter tuning. Given a planned pipeline such as pplan from Figure 2, users access this execution mode by invoking the auto_confinn igure method:
beat_found pplan. cuto__configure train_X, train_y, optimizer Ryypero,rrtypprrot,, cv-3, max_opt__time t00eemm-0m max_eval__t t0me=30)
2
The result (in this case best_ is the trained pipeline that yielded the best score (here, cross- validated with ev-3) during the search. Section 4 discusses the details of the translation.
Schemas. Besides combinators, which specify space of pipelines to search, AutoMI. also needs hyperparameter schemas, which specify a space of hyperparameter values to search. Here, we adopt the value-set definition of a schema (or type) as a space (or set) of values [33]. While data scientists frequently write code with combinators to exercise control, they write schemas less often. As of today, Lale includes schemas for 216 operators, sufficient for many common machine learning tasks.
Figure shows the most important parts of the schema syntax (eliding some details, such as defaults, integers, and array schemas, that are not needed for examples in this paper). The start symbol, schema, can be an enumeration, a range, an object, or it can nest other schemas by using one of three logic connectives. An enum schema specifies a categorical hyperparameter with choice between one or more values. Figure includes multiple examples of enums: [mle]. trueefalsee, [0.25]. [linear, sag, lbfgs), etc. A range schema specifies continuous hyperparameter with a minimum and

---

schema : ::=mmm] range| object
disj
conj neg
PCA: N: (0..1) V [mle]}
enum := [ VALUE* ]
J48 dict [true, false,, C: (0..0.5)
range = < NUM NUM) distr
(dict( R: [true]] + dict(C: [0.25]
distr = uniffrrm" "loguniform"
LR dS [linear, sag, lbfgs] P: 111, 1
object = dict { (NAME: schema)
(dictí(: [sag, lbfgs]} - dict<P: [[2}})
disj = schema v schema
conj == schema A schema
Figure 4: Schema examples for 3 operators (sim- plified): sklearn's PCA and LogisticRogreessoon and Weka's 's J48. We write = t for (-s) Vt.
neg
= schema
Figure 3: Schema syntax.
maximum (most AutoML optimizers do best with bounded ranges) and an optional distribution distr. Figure 4 includes two examples of ranges: (0..1) and (0..0.5). An object schema specifies dictionary of zero or more hyperparameter schemas. For example, Figure 4 specifies one hyperparameter for the PCA operator from sklearn [7] and two each for J48 from Weka [18] and for LR from sklearn.
The schema syntax supports three logic connectives disjunction (V), conjunction (A), and negation (-). Disjunction describes union type: for example, in Figure 4, the N hyperparameter of PCA can specify the number of components with either a range (0..1) or an enum imle). Conjunction describes an intersection type and negation can be used to describe an implication. These connectives help express constraints for ruling out combinations of hyperparameters that are individually valid but collectively invalid. For example, with Weka's J48 operator, if R is true (use reduced error pruning), then C must be 0.25 (cannot set the pruning confidence to other values). Figure 4 specifies this via +, which is syntactic sugar for the basic logic connectives. Hyperparameter constraints are useful for pruning invalid areas from the search space, so AutoML search can focus on valid areas instead.
To make schemas easier to use, we decided not to invent our own new schema language, but instead to adopt a mature existing language with a broad user base and tooling ecosystem. One choice we considered was Python 3 types [45, 37]. Unfortunately, Python 3 types are not expressive enough for specifying constraints. But fortunately, there is another language that also works well with Python and is used extensively by large community well beyond that of AutoML: JSON Schema [35]. Most constructs from Figure 3 have exact equivalents in JSON Schema. For instance, a range (a..b) can be written as ("type": "number", "minimum" :a, "maximum" :b). Lale adds a few extensions to JSON Schema. For instance, it supports an optional "distribution" keyword a "number" " schema; and supports "operator" type for hyperparameters that accept a nested machine-learning operator.
3 Gradual Automation
The previous section described pipeline combinators and hyperparameter schemas. This section builds on that foundation to describe a programming model for gradual AutoML. Here, gradual means that users get to use total automation if they so desire, but can also exercise fine-grained manual control over any aspects of the pipeline. The programming model design was guided by three principles. First, progressive disclosure lets users start by learning just subset of the constructs, only requiring them learn more constructs as needed to accomplish more tasks. Second, orthogonality minimizes the number of independent constructs to make the programming model easy to learn, while maximizing the ways these constructs can be plugged together make the programming model expressive [27]. Third, the principle of least surprise makes each construct look and behave the way most users would expect. That why we started from well-established programming models sklearn [7] and JSON Schema [35] and try to be faithful to their expectations.
Figure show the learning curve of gradual AutoML from zero to mastery. The rest of this section describes each set of constructs, starting at the bottom and working upwards one layer at a time.
Use total automation. Lale provides an AutoPipeline operator. From the outside, it looks and feels just like any sklearn operator with fit and predict methods. Internally, fit runs AutoML on a predefined planned pipeline, which users need not dwell on. Hence, total automation with Lale is trivial to learn for users who already know skleam (least surprise).
Inspect returned pipeline. One level up from total automation is to look at its result: even users who have little interest in controlling the pipeline may want to see for transparency. Lale provides a visualize method and round-trippable pretty_ pprint method that returns pipeline's ''s Python code.

---

mastery
add new optimizers
new search space encoder ssarch point decoder
write pipeline grammans
rocursive operator definition, sample, unfeld
add operntors
me nake_ make_oppeeeeeeeeee operator (irpl, achemas)
control search space details
freese, customise oeteooneeeeeeeeeeeeeeeeeee schema
refine returned pipeline
functions that decompase pipelines
plan pipeline
1 & operalor library, auto cnte_configure
inspect returned pipeline
visualise, peattyprima petoPipeline, rit, predict
use total automation
Figure 5: Gradual automation with progressive disclosure.
Plan pipeline from scratch. Learning about the combinators and the auto auto_conf igure method from Section 2 brings users to the next level. To further assist in using Lale's collection of 216 operators, Lale generates documentation from schemas and uses schemas for error checking [17, 38].
Refine returned pipeline. Data science, even with automation, is often iterative. After an AutoMI. search (whether with full automation or with hand-planned pipeline),
may want to tweak the pipeline for another search. Lale supports this with functions for breaking down pipeline into smaller pieces, which can then be composed back into new pipeline using combinators (orthogonality).
Control search space details. To exercise even more control in a localized and composable way, users can disable AutoML search for sub-pipeline with freeze fweeeeetraineeeeeeeee trainable, or they can disable refitting with freeze_trained eters of operator, such adjusting a range. Given returns a new operator that is the same as & except for the given schema tweaks. Again, these constructs are orthogonal combinators: frozen customized operators can be used like any others.
trained. Furthermore, users may want to locally tweak the schema for some hyperparam-
operator calling customizee act.eeaa__ceee.. customiiee__hhema...
Add new operators. Given an operator without schema, Lale infers simple schema that just sets all hyperparameters to their defaults. Users can then call customize_ _cchema to make them selectively tunable. Or they can write a comprehensive schema and attach it to the operator using _akk__opeaator
Write pipeline grammars. Some AutoML tools [31, 10, 12] explore not a rigid template for pipeline topologies but possibly unbounded space of topologies, including some that define context- free grammar using Lale's 's combinators [20, 28]. Lale lets users define such search spaces by declaring recursive operators, which are self-referential definitions (e.g., g.p g. q). Recursive operators are orthogonal to the other constructs such as combinators, with which they can nest. The methods sample e and unfold turn these into normal Lale pipelines suitable for auto_configure.
Add new optimizers. Only few users get so close to mastery that they desire to add their own optimizers. While the details will vary on case-by-case basis, the next section describes a principled translation scheme that helped Lale target several existing optimizers.
4 Translation Scheme
This paper adds a third execution mode to sklearn-style pipelines: the first two modes are prediction and training, and the third mode is AutoML search. Lale's API for this mode is auto_conf igure, search space for given optimizer and then runs that optimizer.
which first translates the pipeline
Figure details that interaction to illustrate how the translation scheme (encode and decode) interacts with an optimizer and how it uses the first two execution modes (predict and fit). The starting point planned pipeline, which has operators whose hyperparameters need to be tuned or operator choices (I combinator) whose operators need to be selected. Step "encode search space" translates the planned pipeline to an optimizer-specific search space. Once called, the optimizer loops over trials, which are search points it selects based on its solver, e.g., TPE [5]. Step "decode search point" turns an optimizer-specific search point (i.e., a specific value combination in the search space) into configuration needed for instantiating trainable pipeline. In trainable pipeline, all hyperparameters
5

---

and operator choices are bound to specific values. Fitting the trainable pipeline to training data (folds) yields trained pipeline, where the learn- able coefficients of all operators are also bound. The next step is to call predict on the trained pipeline and to score the result to compute a loss for the optimizer. After wrapping up its loop, the optimizer returns the best trial, and configure returns the corresponding trained pipeline.
aulo ronfigure
optimizer
PlannedPipaline
space
optimize
run trial
decode search point
TrainablePipeline
TrainedPippeline
IIt
create
predict
retum loss
The translation scheme is one of the contributions of this paper. Its job is to enable the modular combinator-basee syntax for users, by weaving together the operators and their hyperparame- ter schemas and rewriting them to a
return best trial
TrainedPipeline
Figure 6: Interaction with an optimizer.
form suitable for given optimizer. Users thus make local changes using orthogonal constructs without worrying about their impact on the rest of the pipeline. To target a variety of optimizers, we structured the translation scheme as series of rewriting passes between different forms. Each form has its own syntax restricting how a search space can be expressed. Figure shows the starting form. Each pass is designed to be semantics preseeriig......g......... in the sense that the search space encodes the same set of pipeline instances before and after the pass. The running example for this section is the pipeline LR); Figure 4 shows its schemas in starting form.
(J48
Normalize. Figure shows the normal form, which a disjunction of objects. This form is crafted to simplify later rewriting passes by letting them start from less feature-laden form. While nesting still possible in normal form, it is restricted to objects. Also, once in normal form, schemas contain no conjunctions or negations, which would inhibit local reasoning. Normalization works in bottom-up pass. At each level, it performs disjunction hoisting using rewrites such as De-Morgan's 's law. Negation and conjunction
pushed into primitive schemas. This temporarily give rise uninhabited or trivial nested schemas, which the pass simplifies away with another set rewrites. Our translation scheme normalizes the schema of each operator in a pipeline separately.
schema object (v object)*
PCA dict(N: (0..1)} V dict [mle]}
object == dict (NAME: prop)
J48 dict R:
false). C: (0.005)33
prop enum range object
dict R:
trus,false] C: [0.25]}
enum [ VALUE+
LR dict((S:
linear), P: (1, 12
range = C NUM NUM) dist?
dict(S:
|lieear sag, 1bfgs], P: [2]}
distr = uniforrm "loguniform"
Figure 7: Normalized schemas: syntax (left) and example (right).
Combine. Figure 8 shows the combined form, which represents the entire pipeline together instead of each operator separately. Combining encodes combinators follows. The >> and & combinators turn into an object schema (dict), where each operator becomes step 8;. The combinator turns into a disjunction schema (V), with an additional discriminant D to track what was chosen. Higher-order operators yield nested object schemas (dict). The encoding preserves sufficient information to enable later decoding. After this pass, the encoding is conceptually similar to that of Hyperopt [6], so our implementation provides a simple backend translating it into a Hyperopt search space.
Flatten. Most other optimizers require a less hierarchical search space than the combined form. Figure shows the flattened form, which is a disjunction objects without further nesting. The flatten pass hoists properties from nested objects, renaming them as needed for uniqueness (in way that enables later decoding). The flattened form is close to what SMAC [15] needs, except that objects of different disjuncts have different properties. In SMAC, all disjuncts have the same properties, with

---

so: (icc(N:(OO..))) dict [mle]}
space =
object
dict-(D: [J48] R: [false), C: (0..0.5)}
object =
dict (NAME: prop)
dict
dictCD: J48]. R: truue C: [0.25]]
prop = enum range disj
$1;
diet D: [R), S: [inear], P: [1 12 V
disj
object (v object)
dict(D: [R), St
[linear,sae sag. lbfgs). P: [12]]
Figure 8: Combined search space: syntax (left) and example (right).
dict[N: (0..1), , D: J48] R: false],
C: (0..0.5)
V dict([N: (0..1), D: J48 R: true.false
C:
dict(N: mmle] D: J48 R:
false);
C: (0..0..5)
space :- object (V object)*
dict((N: [mle], D: [J48], R: True.false
[0.25] }
object = dict { (NAME: prop)" }
dict(NN: (0..1), D: [LR], S:
[lneari
P: [1, }
prop = enum range
dict<N: N:
(0..1), D: [LR] S:
linear, sug, 1bfgs),
P: [12]
V dict- N:
mle]. D: LR S:
linear),
P: [1, 12]
}
dict- N: mlee, D: [.R). S:
|linear, sug, lbfgs),
P: [12)
Figure 9: Flattened search space: syntax (left) and example (right).
conditionals on the side indicating which variables are relevant. The SMAC backend thus generates these conditionals, and renders the space in SMAC's parameter configuration space format [42].
The ADMM backend uses a partially flattened form. ADMM [26] supports a pipeline of operator choices. The backend thus leaves the top-level spine of pipeline intact pipeline of choices. If one of the choices contains a complex operator (e.g. a nested pipeline), flattened as for SMAC.
Discretize. The optimizers GridSearchC' and HalvingGriddSearchCV from sklearn [7] require continuous ranges be made categorical. Discretization for a hyperparameter first includes the default and then samples a user-configurable number of additional values from the range based on the prior distribution. Figure 10 shows the discretized form when generating two values per range.
dict{N; [0.50, 0.01], D: [488), R: [false], C:: [0.25, 0.01]}
diet-CN: [0.50. 0.01 D: [J48]. R: (true, C: |0.25]]
space == object (v object)* *
V dict(N:: [mle], D: [488]. R: [false), C: [0.25, 0.01]
object = dict( (NAME: enum)
V
dict N: [mle], D: [LRR, [linear, sag, 1bfgs), P: [(]]}
Figure 10: Discretized search space: syntax (left) and example (right).
Results
This section presents experiments evaluating Lale's usability and expressiveness. While the experi- ments focus on tabular data and do not use deep learning, Lale also works with other data modalities such text [8], code [29], and time-series [39], including with pipelines that contain deep-learning operators.
RQI: How usable is gradual AutoML with combinators?
To explore this research question, we compare Lale to the popular sklearn machine learning framework in user study with 18 participants. The study design is between-subjects [9], where participants are randomly assigned to a Lale or sklearn version of the study. We asked participants to perform four tasks involving sample machine learn-
Project
Normalizer
Cuncal-
K- Neighbars- Classiffer
Solect-
Project
Best
Figure 11: Sample pipeline used in user studies.
ing pipelines: (T1) inspecting and understanding an existing pipeline, (T2) refining the given pipeline by replacing operators and hyperparameters, (T3) debugging a failing pipeline, and (T4) searching across a space potential pipelines. The tasks are based on the principle of progressive disclosure in that we provide sample pipeline (Figure 11) using common manual constructs in either sklear or
7

---

Lale format and ask participants to perform tasks that use progressively expanding set of constructs. All participants self-reported having at least some sklearn experience, worked at IBM, and were not further compensated for their voluntary participation. The supplemental material describes the detailed study design and methodology, including the sklearn and Lale notebooks used.
One surprising result of the user study is that four (44%) of the sklearn participants were not able to correctly identify the branching structure of the preprocessing subpipelines of the sample pipeline during Tl (seen in Figure 11), compared to a single Lale participant (11%). Two of the Lale participants also used inspection features in Lale such as the vieualize and pretty-priws methods. In T4, two (22%) of the sklearn participants were either unable to correctly (or at all) implement some method of searching across a space of potential pipelines whereas all of the Lale participants were able to complete the task. When implementing T4, most of the Lale participants (6 or 67%)
used the or combinator (1) and the AutoMI. execution mode to perform automated search as opposed to more manual methods. This is compared to two (22%) sklearn participants using automated search via GridSearchCV. The relatively high success rate of Lale participants both in understanding existing pipelines that use the and combinator (4) and in composing new planned pipelines that use the or combinator (1) suggests that new Lale users are able to pick up and use these independent constructs. This supports the principle of orthogonality in the Lale programming model. Automated search also tends to be more concise than manual implementations for the search task, perhaps contributing to Lale participants (59% less) compared to sklearn participants. Lale participants also on average performed all of the tasks faster than sklearn participants. over 2.5 minutes faster total (13% faster). (In reality, the time difference would likely be even higher, since we stopped participants who take a manual approach early, see supplement for details.) For T2 and T3, participants performed equally well in both versions of the study. We note that some Lale participants were able to refine pipelines without referring to documentation, relying on their sklearn knowledge replace operators and define their hyperaarameters. This supports the principle of least surprise in that Lale operators work in ways that experienced sklearn users expect. Table summarizes the results from the user study.
average implementing the task in 25 fewer lines of code (LoC)
Table 1: Summary of user study results.
VERSION
COUNT
TI CORRECT
T4 CORRECT
T4 LoC MEDIAN (ALL))
TOTAL TIME MEAN (STDDEV)
Lale
9
89%
100%
10
8,9,8 9, 10, 10, 10, 75)
17:36 (5:03)
sklearm
56%
78%
23 15, 16, 22, 24, 43, 100, 100)
19:26 (6:34)
Total
18
72%
89%
14
18:54 (5:45)
Answer to RQ1: In our user study, gradual AutoML with combinators was usable enough to increase task correctness by more than 20% while reducing lines of code by more than half.
Does the translation scheme work for diverse planned pipelines? To explore this research question, we used Lale's auto_configury on several planned pipelines with Hyperopt and its TPE solver. We used auto-sklearn [15] as a state-of- art baseline; it uses SMAC for joint algorithm selection and hyperparameter tuning similar to our work, but also uses meta-learning, which we do not yet do. We used a 2.0GHz virtual machine with 32 cores and 128GB memory and gave each search 1 hour time budget with a timeout of 6 minutes per trial, which corresponds to the default setting of auto-sklearn. We chose 14 datasets from OpenML [46] (CC-BY license) that allow for meaningful optimization (as opposed to just the initial few trials) within that 1-hour budget. Our datasets are drawn from the AutoML Benchmark [16] and four of them were also used in the auto-sklearn paper [15]. (The goal of our evaluation is orthogonal to the sizes of the datasets.) We used a 66:33% % train:test split with 5-fold cross validation on the train set during optimization.
RQ2:
Table 2 summarizes the results. Column AUTOSKL shows the auto-sklearn baseline; LALE-AUTE AUTO uses Lale's 's AutoPipeline operator from Section 3; LALE-AD3M and LALE-TPOT use Lale grammars inspired by AlphaD3M [12] and TPOT [31] unfolded with a maximal depth of 3; LALE ADB uses pipeline with a choice of preprocessors and the auto-sklearn [15] with the set of classifiers limited to AdaBoost with only preprocessing. (See supplement for details.) All searches succeed and most yield performance in the same ballpark as auto-sklearn (aside from shuttle, which triggers an auto-sklearn issue that other users have also encountered). Considering all 14 datasets, LAL.E-UTO performs on average 0.7% better than AUTOSKL.. Excluding shuttle, Lale performs on average 0.3% worse than AUTOSKI. These 0.3% are smaller than the standard deviation on most datasets, and based on a t-test, they are not statistically significant.
higher-order operator; and ASKL-ADB
8

---

Table 2: Test accuracy for 14 OpenML classification tasks, using 4 Lale search spaces with Hyperopt.
Absolute accuracy mean (and stddev) over 5 runs
100 x (LALE/AUTOSK) 1)
LALE-AUTO LALE-TPOT
LALE-AD3M
ASKL-ADB
AUTO
TPOT
AD3M
australian
85.1(0.4)
86.2
85.9 (0.6)
86.8 (0.0)
(1.0
(3.1)
1.3
0.9
blood
77.9114)
(0.3
77.5 (2.5)
74.7 (0.7)
77.1 (0.7)
74.7 (0.8)
-3.3
-0.5
-4.0
-1.0
breast-cancer
73.0(0.6)
73.0 (0.0)
(1.4
(3.3)
70.9 (2.0)
72.4 (0.5)
0.0
-2.3
4,9
-2.9
99.400.1)
97.7
(0.3)
-6.7
-1.1
credii-g
76.6(1.2)
75.7 (0.0)
74.1 (0.5)
74.8 (0.4)
(1.3)
76.2
-1.1
-3.2
-2.4
-0.7
diahetes
77.0(1.3)
(1.4
(0.2)
75.0
-0.9
jungle-chess
88.1().2)
(2.4
82.9
74.1 (2.0)
(2.4
86.9
4.9
0.9
-15.8
1.5
83.8(0.3)
(3.0
83.5
80.6
(0.4
(0.3)
-0.5
-0.4
-0.2
-0.6
kr-ve-pp
99.7(0.0)
(0.)
(9.0
96.8
(0.5
(0.5(
-0.2
-2.9
-0.2
mfcat-factors
98.7(0.1)
97.1 (0.0)
97.9
(0.5
97.5 (0.4)
97.9 (0.1)
-1.6
-1.5
-1.2
-1.2
phoneme
90.3(0.4)
(9.6
(0:4)
-0.2
shutle
87.3(116))
100.0 (0.0)
(9.9
(9.9
10.00
(0.00
14.5
14.5
14.4
14.6
spectf
87.9(0.9)
87.7 (0.0)
(2.2)
(6.6
(2.6)
89.7 (2.9)
-49
sylvine
95.4(0.2)
(0.0)
94.4 (0.7)
91.3 (0.1)
95.1 (0.2)
95.1 (0.1)
-0.4
-1.1
-.3
-0.3
Answer to RQ2: The translation scheme works for pipelines with complex topologies. including those generated by grammars inspired by AlphaD3M and TPOT, well as pipelines including ensembles, producing competitive results.
RQ3: Does the translation scheme work for diverse optimizer backends? To explore this research question, Table lists test accuracies using 9 different optimizers for the OpenML phoneme dataset. The planned pipeline uses scaler, polynomial feature generator, and choice between XGBoostClassifier and LGB.Classifier. The other experimental settings are the same as above. All translations succeed, and most yield reasonable accuracy. except for the ones that timed out. The translated search spaces for GridSearchCV and HalvingGridSearchee explore within the time budget. Most people would only grid-based search for very small search spaces, but then, suffer from exploring the space less thoroughly than "proper" AutoML tools.
were correct but too large to
Table 3: Test accuracy (mean and stddev) for the phoneme dataset using multiple optimizer backends.
Hyperopt Anneal
Hyperband
ADMM BOBa
GridSearchCC
Halving-
TPE
Rand
RND
GPRND
GrdSSerccCCV
88.75 (0.2)
88.94
88.75 (0.5) 87.94 (0.2)
89.00 (0.2)
(0.2)
88.61 (0.3)
Timed
Timed
Answer to RQ3: The translation succeeds for Hyperopt (3 optimizers), Hyperband, ADMM (3 opti- mizers), GridSearchCV, and HalvingGridSearchCyy
6 Related Work
Combinators, which were discovered in the 1920s [41], have long been prominent in functional programming languages such as Lisp [43]. The advantage of their tacit style has been recognized more broadly; for instance, the Unix pipe [30] can be viewed as combinator. Combinators have also enabled some sophisticated innovations such as parser combinators [23].
Combinators and data science. MapReduce [11] is successful programming model for large- scale data processing named after two combinators [22]. Allison [2] describes Haskell combinators for converting between different machine-learning model types, but does not diseuss AutoML. Sklearn [7] provides combinators muake_pipeline and make_unioz) as well as some basic AutoML (GridSearchCv, KalvingGridSearrd iSSarchhV, and BandomizedSearchc)) but only integrates combinators with AutoML loosely, instead requiring users to perform manual name-mangling and non-compositionall repetition. Lale builds on sklearn, extending and streamlining its combinators and tightly integrating them with AutoML. Pilat et al. [36] describe a combinator language suitable as a target for automatic code generation with genetic programming, but intended be written by hand. Unlike our work. they focus on total (not gradual) automation and do not provide an or combinator (1).
Auto programming models. AutA demonstrated how to support conditional hyper- parameters multiple solvers, but the paper focused on single planned pipeline, not gradual automation [44]. While both hyperopt-sklearn [21] and auto-sklearn [15] support some gradual automation, instead of tightly integrating gradual automation with combinators, they require users to

---

drop to a bespoke lower-level API: Hyperopt for hyperopt-sklear and SMAC's parameter configura- tion space format [42] for auto-sklearn. We argue that JSON Schema [35]. being a language with a track record of wide-spread use by programmers who are not AutoMI. experts is less low-level; e.g., it is the backbone of OpenAPI [32]. We also built tool to extract schemas from sklearn docstrings [4]. In Optuna, users write imperative code that first samples values, then uses either if-statements for operator choice or passes values to constructors for hyperparameter tuning [1]. In contrast, combinators are more concise, more localized, and less surprising to sklearn users. OBOE [48] proposes an alternative meta-learning approach from that of auto-sklearn; requires an initial discrete set of models with hyperparameters choose from, and Lale could potentially be used to specify this set of models. In PyGlove, users start from a manually written neural network which they then "hyperify" by replacing a component or hyperparameter with "hyper value" [34]. Their notion of "hyper" is similar to our notion of "planned", but they do not use combinators, and they foeus only on neural architecture search. AutoGluon-Tabula focuses total automation and the paper does not discuss gradual automation [13]. In AutoGOAL, users specify a hierarchy of pre-defined "indicator" types, then the system tries any pipeline of operators that gets from the input to the output [14]. AutoGOAL's API deviates from sklearn conventions, and while it uses the syntax of Python types, it does not capture constraints, nor is it compatible with Python 3 type checkers.
7 Limitations and Societal Impacts
limitation of the user study is that it used only 18 participants for the sklearn treatment plus 9 different participants for Lale), and that all participants worked at IBM. Another potential limitation that researcher bias may be present in the design of the user study tasks. We designed the tasks to showcase and validate aspects of the Lale programming model which may potentially bias tasks in favor of Lale. We attempt to mitigate this by only selecting participants who are experienced in sklearn and by only allowing Lale newcomers to participate in the Lale version of the study.
One limitation of the translation scheme that not guaranteed to handle all JSON Schema features; in fact, we encountered and fixed some limitations early during development. Fortunately, there were no new limitations recently, and there theoretical work that could be used to tackle them if they arise [3]. A limitation of schema-base interdependence between operators. We mitigate that by providing not just hyperparameter schemas but also dataset schemas, which enable us to check whether data flowing from one operator to another in pipeline compatible [17]. A limitation of the programming model is that, while achieves a high level of sklearn compatibility, Python code that explicitly reflects over dynamic types still expose differences. We mitigate this by testing that Lale interoperates well with a broad set of sklearn features including metrics, -ross-validation, operators, etc.
constraints is that they currently do not capture
One potential societal impact is that enabling automation of searching machine learning pipelines may encourage computationally--ee On the other hand, we hope gradual automation will encourage data scientists use their domain knowledge to more selectively and efficiently automate their machine learning work. In future work, we are planning to adopt caching techniques [24, 47] to further reduce CO2 emissions.
approaches to data science tasks and raise CO2 emissions.
Conclusion
This paper presents combinators for gradual AutoML, along with novel translation scheme from combinators to existing optimizers. We have implemented our combinators in an open-source library called Lale, while ensuring they interoperate smoothly with the sklearn ecosystem. Besides being open-source, Lale is also actively used in a commercial AutoMI, product, highlighting the real-world value of the underlying concepts. Our ultimate goal is to make AutoML accessible manual ML. by offering a gradual combination of both.
10

---

References
[1] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: next-generation hyperparameter optimization framework. In Conference on Knowledge Discovery and Data Mining (KDD) Applied Data Science Track, pages 2623-2631, 2019.
[2] Lloyd Allison. Models for machine learning and data mining in functional programming. Journal on Functional Programming (JFP), 15(1):15-32, January 2005.
[3] Nohamed-Amine Baazizi, Dario Colazzo, Giorgio Ghelli, Carlo Sartiani, and Stefanie Scherzinger. Not elimination and witness generation for JSON Schema. In Conférence sur la Gestion de Données (BDA), October 2020.
[4] Guillaume Baudart, Peter Kirchner, Martin Hirzel, and Kiran Kate. Mining documentation extract hyperparameter schemas. In ICML Workshop on Automated Machine Learning (AutoML@ICM)) 2020.
[5] James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper- parameter optimization. In Conference on Neural Information Processing Systems (NIPS),
pages 25544,554, 2011.
[6] James Bergstra, Brent Komer, Chris Eliasmith, Dan Yamins, and David D. Cox. Hyperopt: Python library for model selection and hyperparameter optimization. Compulational Science & Discovery, 8(1), 2015.
[7] Lars Buitinek, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake VanderPlas, Arnaud Joly, Brian Holt, and Gaël Varoquaux. API design for machine learning software:
Experiences from the scikit-learn project, 2013. https://arxiv.orr/abbi
309.0238.
[8] Arunima Chaudhary, Alayt Issak, Kiran Kate, Yannis Katsis, Abel N. Valente, Dakuo Wang, Alexandre V. Evfimievski, Sairam Gurajada, Ban Kawas, A. Cristiano I. Malossi, Lucian Popa. Tejaswini Pedapati, Horst Samulowitz, Martin Wistuba, and Yunyao Li. AutoTeex: An end- to-end AutoAl framework for text. In Demonstration at Conference Artificial Intelligence (AAAI-Demos), pages 16001-16003, 2021.
[9] John W. Creswell. Research design: Qualitative, quantitative, and mixed methods approaches. SAGE publications, 4th edition, 2013.
[10] Alex G. C. de Sá, Walter José G. S. Pinto, Luiz Otavio V. B. Oliveira, and Gisele L. Pappa. RECIPE: A grammar-based framework for automatically evolving classification pipelines, In European Conference on Genetic Programming (EuroGP), pages 246-261, 2017.
[11] Jeffrey Dean and Sanjay Ghemawat. MapReduce: Simplified data processing on large clusters. In Symposium on Operating Systems Design and Implementation (OSDI), pages 137-150, 2004.
[12] Iddo Drori, Yamuna Krishnamurthy, Raoni Lourenco, Remi Rampin, Kyunghyun Cho, Claudio Silva, and Juliana Freire. Automatic machine learning by pipeline synthesis using model-based reinforcement learning and grammar. In ICML Workshop on Automatic Machine Learning (AutoML (NtoML@ICML).. 2019.
[13] Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhangh, Pedro Larroy, Mu Li, and Alexander Smola. AutoGluon- Tabularr Robust and accurate AutoML for structured data. In ICML Workshop on Automated Machine Learning (AutoML @ ICML), 2020.
[14] Suilan Estevez-Velarde, Alejandro Piad-Morfiss, Yoan Gutiérrez, Andrés Montoyo, Rafael Munoz, and Yudivian Aimeida-Cruz, Solving heterogeneous automl problems with AutoGOAL. In ICML. Workshop on Automated Machine Learning (AutoMI. @ ICMI.), 2020.
[15] Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg. Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. Conference on Neural Information Processing Systems (NIPS), pages 2962-2970 2015.

---

[16] Pieter Gijsbers, Erin LeDell, Janek Thomas, Sebastien Poirier, Bernd Bischl, and Joaquin Vanschoren. An open source AutoML benchmark. In ICML Workshop on Automatic Machine Learning (AutoMI.C @ ICML), 2019.
[17] Andrew Habib, Avraham Shinnar, Martin Hirzel, and Michael Pradel. Finding data compatibility bugs with JSON subschema checking. In International Symposium on Software Testing and Analysis (ISSTA), pages 620-632, 2021.
[18] Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H. Witten. The WEKA data mining software: An update. SIGKDD Explorations Newsletter, 11(1):10-18, November 2009.
[19] Martin Hirzel, Kiran Kate, and Parikshit Ram. Engineering fair machine learning pipelines. In ICLR Workshop on Responsible Al TRAI@ICLR)) 2021.
[20] Michael Katz, Parikshit Ram, Shirin Sohrabi, and Octavian Udrea. Exploring context-free languages via planning: The case for automating machine learning. In International Conference on Automated Planning and Scheduling (ICAPS), pages 403-411, 2020.
[21] Brent Komer, James Bergstra, and Chris Eliasmith. Hyperopt-skleam: Automatic hyperpa- rameter configuration for scikit-learn. In Python in Science Conference (SciPy), pages 32-37, 2014.
[22] Ralf Lämmel. Google's ' MapReduce programming model revisited. Science of Computer Programming, 70(1):1-30 2007.
[23] Daan Leijen and Erik Meijer. Parsec: Direct style monadic parser combinators for the real world. Technical Report UU-CS-2001-27, Universiteit Utrecht, 2002.
[24] Liam Li, Evan Sparks, Kevin Jamieson, and Ameet Talwalkar. Exploiting reuse in pipeline- aware hyperparameter tuning, 2019. https//aarix.orr/ab///99......
[25] Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hy- perband: A novel bandit-based approach to hyperparameter optimization. Journal of Machine Learning Research (JMLR), I 18(155):1-5,, 2018.
[26] Sijia Liu, Parikshit Ram, Deepak Vijaykeerthy, Djallel Bouneffouf, Gregory Bramble, Horst Samulowitz, Dakuo Wang, Andrew Conn, and Alexander G. Gray. An ADMM based framework for AutoML pipeline configuration. In Conference on Artificial Intelligence (AAAI), pages 4892-4899, 2020.
[27] B.J. Mailloux J.E.L. Peck, and C.H.A. Koster. Report on the Algorithmic Language ALGOL 68. Springer, 1969.
[28) Radu Marinescu, Akihiro Kishimoto, Parikshit Ram, Ambrish Rawat, Martin Wistuba, Paulito P. Palmes, and Adi Botea. Searching for machine learning pipelines using context-free grammar. In Conference on Artificial Intelligence (AAAI), pages 8902-8911, 2921,
[29] Georgios Mavroudeas, Guillaume Baudart, Alan Cha, Martin Hirzel, Jim A. Laredo, Malik Magdon-Ismaill Louis Mandel, and Erik Wittern. Learning GraphQL query cost. In Conference on Automated Software Engineering (ASE) Industry Track, 2021.
[30] M. Douglas Mcllroy. A research UNIX reader: Annotated excerpts from the programmer's manual, 1971-1986. Technical report, Bell Labs, 1987. https://www..s..armmoth..cmm reader,pdf.
[31] Randal S. Olson, Ryan J. Urbanowicz, Peter C. Andrews, Nicole A. Lavender, La Creis Kidd, and Jason H. Moore. Automating biomedical data science through tree-based pipeline optimization. In European Conference on the Applications of Evolutionary Computation (EvoApplicatioons)) pages 123-137, 2016.
[32] OpenAPI Initiative, OpenAPI specification (fka swagger restful api documentation specifica- tion). 2014. http://///uhcnnnoooooooooootsttt
12

---

[33] David L. Parnas, John E. Shore, and David Weiss. Abstract types defined as classes of variables. In Conference on Data: Abstraction, Definition and Structure, pages 149-154, 1976.
[34] Daiyi Peng, Xuanyi Dong, Esteban Real, Mingxing Tan, Yifeng Lu, Gabriel Bender, Hanxiao Liu, Adam Kraft, Chen Liang, and Quoe Le. PyGlove: Symbolic programming for automated machine learning. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[35] Felipe Pezoa, Juan L. Reutter, Fernando Suarez, Martín Ugarte, and Domagoj Vrgoë. Foun- dations of JSON schema. In International Conference on World Wide Web (WWW), pages 263-273, 2016.
[36] Martin Pilat, Tomas Kren, and Roman Neruda. Asynchronous evolution of data mining workflow schemes by strongly typed genetic programming. In International Conference on Tools with Artificial Intelligence (ICTAI), pages 577-584, 2016.
[37] Ingkarat Rak-ammouykit, Daniel McCrevan, Ana Milanova, Martin Hirzel, and Julian Dolby. Python types in the wild: tale of two type systems. In Dynamic Languages Symposium (DLS), pages 57-70, 2020.
[38] Ingkarat Rak- aakkuunnuy,i,, Ana Milanova, Guillaume Baudart, Martin Hirzel, and Julian Dolby. Extracting hyperparameter constraints from code. In ICLR Workshop on Security and Safety in Machine Learning Systems (SecML ICLR), 2021.
[39] Subhrajit Roy, Kiran Kate, and Martin Hirzel. A semi-supervised deep learning algorithm for abnormal EEG identification. In Machine Learning for Health Workshop (MLAH @ NeurlPS), 2019.
[40] Chirag Sahni, Kiran Kate, Avraham Shinnar, Hoang Thanh Lam, and Martin Hirzel. RASL: Relational algebra in scikit-learn pipelines. In Workshop on Databases and A/ (DBAI@ NeurlPS), 2021.
[41] Moses Schönfinkel. Über die Bausteine der mathematischen Logik. Mathematische Annalen, 92(3):305-33 316, 1924.
[42] SMAC Parameter Configuration Space (PCS). http://aataml.gghhaaioossA/C/Cst mmmllpprameter-cofffgurtaan spacon--sppppsssssssees pcs.
[43] Guy L. Steele, Jr. Common Lisp: The Language. Digital Press, 2nd edition, 1990.
[44] Chris Thornton, Frank Hutter, Holger H. Hoos, and Kevin Leyton-Browe Brown. Combined selection and hyperparameter optimization of classification algorithms. In Conference on Knowledge Discovery and Data Mining (KDD), pages 847-855, 2013.
[45] Guido van Rossum, Jukka Lehtosalo, and Lukasz Langa. PEP 484 Type Hints, 2014. https: www.ppytton..rrgdee/pppppppppppp 0484/.
[46] Joaquin Vanschoren, Jan N. van Rijn, Bemd Bischl, and Luis Torgo. OpenML: Networked science in machine learning. SIGKDD Explorations Newsletter. 15(2):49 60, June 2014.
[47] Doris Xin, Litian Ma, Jialin Liu, Stephen Macke, Shuchen Song, and Aditya Parameswaran. Helix: Accelerating human-in-the intten-----te-oooop machine learning. In Conference on Very Large Data Bases (VLDB), pages 1958-1961, 2018.
[48] Chengrun Yang, Yuji Akimoto, Dae Won Kim, and Madeleine Udell. OBOE: Collaborative filtering for AutoMI. initialization. In Conference on Knowledge Discovery and Data Mining (KDD), pages 1173-1183, 2019.
13

---

Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 7.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 7.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See supplemen- tal material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5.
(c) Did you report error bars (e.g., with respect to random seed after running experi- ments multiple times)? [Yes] See Section 5, which reports standard deviations after running experiments times.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplemental material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] See Section 5.
(b) Did you mention the license of the assets? [Yes] See Section 5.
(c) Did you include any new assets either in the supplemental material
URL?
[Yes] The Lale library is open-source at https://github. httpt/u..ooomb///ll////// In addition, the supplemental material includes a copy of the code, based on snapshot from the time when we ran the experiments for this paper.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A] We used open datasets and open-source software.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] We used standard datasets from OpenMI. [46] that contained neither personally identifiable information nor offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] See supplemental material.
(b) Did you deseribe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A] There were no risks to participants.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A] Participants were volunteers and did not get compensated for joining our study.
14


Reviews:
[{"review_id": "n7lwRqyQvKf", "reviewer": "Reviewer_6TXR", "summary": "This paper presents a system for gradual AutoML within sklearn. The system focuses on _gradual_ automation, as opposed to total automation, so that the data scientist can incorporate their human intuitions into the AutoML process, saving time and compute resources. The paper introduces combinators that enable developers to write learning pipelines in sklearn, possibly with choices in the model architecture that the AutoML system can search over. There are only 3 combinators (pipe, and, or), their function is easy to understand, and they can be combined and nested arbitrarily. While combinators describe a space of pipelines to search over, the paper also uses an extension of JSON Schema to describe a space of hyperparameters to search over. The paper describes how the schemas can be transformed into formats that are accepted by existing AutoML algorithms. The authors also perform a user study investigating how easily their proposed system can be used, and perform empirical studies comparing how well their AutoML system performs compared to other systems on many datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Positives:**\n* Important problem\n* Easy-to-use system for gradual AutoML\n* Very clear writing\n* Well-thought-out arguments for the benefit of gradual automation and how Lale enables this\n\n**Negatives:**\n* Limited novelty\n* Conclusions from the experiments are not laid out as clearly as I would like\n\n---\n\nThis paper tackles an important problem: creating a framework in which users can use AutoML technologies easily, with fine-grained control over the search space of model architectures and hyperparameters. The proposed system appears to achieve this goal well. It is easy to understand how to use the three combinators to construct pipelines, possibly with different choices in them that give rise to a space of pipelines to search over. It is also easy to use their proposed schemas to specify a space of hyperparameters as well. Altogether, the authors’ Lale library seems like a great way to enable gradual AutoML for users already familiar with sklearn -- I was able to understand the system quickly, and from the user study, it appears that other new Lale users were able to pick it up easily as well.\n\nI am not sure whether the translation scheme from Lale syntax (combinators and schemas) to a form that can be given to an optimizer is as novel as the authors claim. Of course, given that Lale is new, the implementation details of the translation scheme can be considered new as well, but the core ideas in the translation scheme seem standard. The normalize step is quite common when dealing with conjunctions and disjunctions. The combine step is straightforward as well, simply putting the schemas into the hierarchical pipeline structure defined by the combinators. Once we have a hierarchical nesting of search spaces, flattening and discretizing are “obvious” steps to take if a particular optimizer requires a flattened and/or discretized representation. It is good that the paper describes these steps in detail, and providing such detail can count as a contribution of the paper, but I would not describe the translation process as “novel”.\n\n**Originality: 2/5** The authors came up with a well-designed and easy to use syntax for enabling gradual AutoML in sklearn, although the ideas behind this syntax (the pipe/and/or combinators, schemas for hyperparameters) are quite straightforward (which is a good thing for usability!). It also seems that the implementation details to make their syntax work, i.e., the translation scheme, are not completely original either. Ultimately though, it appears that Lale is the first successful (easy to learn and understand, performs well, used in practice) framework for AutoML that naturally supports gradual automation (is this true?), with its usage in industry as further evidence of its success. In my opinion, accounts of such “success stories” are important research contributions, even if their technical details are not entirely original.\n\n**Quality: 4/5** The proposed Lale system itself is technically sound. My only concern is about the empirical results, where I would like more detailed analysis of the results, what conclusions we can draw, and discussion about why those conclusions make sense. See my questions below.\n\n**Clarity: 5/5** The paper was a pleasure to read. The ideas are organized and communicated very effectively.\n\n**Significance: 4/5** This work is quite significant, since it enables more users to easily use gradual AutoML, which will then improve whatever work those users end up doing. I am unsure about the significance of the empirical results though, since Lale appears to perform slightly worse than the SOTA auto-sklearn. \n\n## Questions\n* It is currently not clear whether Lale produces better or worse accuracies than auto-sklearn -- the paper simply mentions that they are in the same ballpark. It seems that Lale gets a slightly worse accuracy on a majority of tasks, but performs much better on the shuttle dataset. I’m assuming that auto-sklearn simply has a bug/flaw that manifests on the shuttle dataset, and the nature of that bug/flaw is unrelated to this paper. Ignoring the shuttle dataset, Lale appears to perform worse than auto-sklearn overall -- why would this be the case?\n* Why would GridSearchCV and Halving-GridSearchCV time out in Table 3? Does Lale produce too large of search spaces for those methods? How do other approaches deal with GridSearchCV?\n\n## Other suggestions\n*  In Table 2, it would be nice to have stats aggregated over all datasets, so there is one number for comparison between the different AutoML methods. However, doing this fairly is tricky. For instance, Lale dominates in performance on the shuttle dataset, so simpy using a mean of the numbers in a column would be skewed toward that result.\n* Since auto-sklearn is the primary baseline for comparison, the paper would benefit from a short description of how auto-sklearn works and the differences between it and Lale.\n* Line 368: interoperates -> interoperate\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors' Section 7 is more than adequate.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "aagZCLuD4YD", "reviewer": "Reviewer_sNtW", "summary": "In the paper „Pipeline Combinators for Gradual AutoML“ the authors propose a machine learning library that allows for convenient definition and usage of ML pipelines. In particular, the library supports the configuration of a pipeline to be (partially) automated.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I very much enjoyed reading the paper since the structure and the language of the paper are excellent.\n\nThe ideas on which the library is based seem to be very targeted and helpful for using it in practice.\nIn the paper the authors give sufficient detail about their library and the specific features that distinguish their library from others such as scikit-learn. Everything is explained in much detail and the architecture and the design appears to be sound.\n\nHowever, in some respects the paper is very technical (due to the nature of its contribution, of course) but it could be less of the perspective of software engineering and focus more on the machine learning / data science part.\n\nSpeaking about the machine learning part, and thus, about the users of the library, I appreciate that the authors carried out a user study. However, I have sone concerns about the study itself:\n\n- First, it is stated that all subjects are from the same company. Guessing that the library is developed by the same company this might mean some bias towards the company‘s product. Be it due to loyalty or due to the library being implemented according to the company‘s principles and philosophies.\n\n- Second, it is stated that all subjects of the study are experienced users of scikit-learn. All the more I was very surprised that several  subjects couldnt identify the right pipeline shape. How is „experienced“ defined? I would have expected that at least reading code should be easy.\n\n- Third, the user base is relatively small in order to reliably draw conclusions based on the acquired data. This limitation should be mentioned somewhere. \n\nTo sum up, I believe that the library is a valuable contribution to the community and I hope that acceptance by users will be reasonable. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The number of subjects in the user study is rather small and homogeneous, which also means a limitation to the generalizability of the results. This should be mentioned somewhere.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "_WlYhUm6n90", "reviewer": "Reviewer_m9sr", "summary": "This work presents a software system for AutoML based on ML pipelines that:\n+ Uses 3 \"combinators\" to construct pipelines.\n+ Abstracts AutoML search using a \"translation scheme\".\n\nA second part of this work focuses on user adoption, backed by usability studies.\nThe user study described in the supplementary material is valuable.", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The contribution of this work as an AutoML system, to the automated composition of complex models, is incremental.\nThis work may benefit by focusing on the user study, specifically on human-model interaction.\n\nThe work is missing:\n- Comparisons with other types of AutoML systems that are not pipeline oriented, such as\n* AutoGluon which focuses on generating complex ensembles and stacking\n* OBOE which focuses on collaborative filtering and fast recommendations.\n- Computation times of the system.\n\nThe work may reference the D3M project which is composed of three tiers:\n(i) TA1 ML primitives, (ii) TA2 AutoML systems, and (iii) TA3 human in the loop and user studies.\nThe paper is missing multiple references to work on the interface between (ii) and (iii) and systems which focus on user studies:\nSee https://datadrivendiscovery.org/d3m-interfaces\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "N3FbrPaSPZX", "reviewer": "Reviewer_2UcV", "summary": "The paper proposes a framework and library of operators for enabling so-called gradual automl. The operators share similarities and extend scikit-learn's methodology of creating ml pipelines. Operators can be nested, creating complex constructs, which can be automatically translated into automl search spaces. The hyperparameters are defined by schemas.\n\nThe library is evaluated in terms of a user study with 18 participants, testing for pipeline understanding, construction and improvement following the introduced progressive disclosure concept. The evaluation also shows empirical results on 14 OpenML classification datasets for available automl algorithms in the framework compared to standard auto-sklearn, showing competitive performance. Finally, the authors also show that different optimizer backends, such as HyperOpt, can be used in the library.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive factors are ...\n- Operators could be an intuitive approach to modeling pipelines\n- Conceptual differences to other libraries are made clear\n\nNegative factors are ...\n- Concept of gradual AutoML not fully convincing, experimental results do not fully back claims\n\nThe paper thoroughly introduces the concept of available/implemented combinators, execution modes as well as the developed schema. The design of the framework is qualitative, elegantly enabling to model complex pipelines in nested structures. \n\nThe authors also provide a detailed related frameworks and libraries, as well as key differences in design decisions. The clarity of the framework components and differences to related works is therefore high. The originality of the framework is also sufficiently high, as it gives practical value to AutoML end-users. \n\nTo this end, I have problems to grasp the \"gradual\" aspect of the approach, which is part of the key message of the paper. The concept is introduced and explained, and a mapping to individual framework functionalities is available, but the latter merely enable manipulation on different abstraction levels. It would be beneficial to clearly state the novelty here, if it is a central claim.\n\nThat said, while the paper maps different gradual AutoML stages to library functionalities, the evaluation does not attempt to compare them to frameworks other than scikit-learn. Why was no other library, as described in the related work section, chosen in addtion? scikit-learn is surely a good baseline, but comparing it to other libraries which enable easier / more interactive modeling might help to support the claims.\n\n### Update after author response\nI thank the authors for their answers to my questions for which I do not have any follow-ups. After reading the other reviews and discussions, I raised my score to a weak accept, as I agree that the paper does not need to show superior performance to other AutoML tools. I still believe that it would be beneficial for the venue to better quantify the added value of the proposed \"gradual\" property, e.g. in the user study. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors address both conceptual limitations as well as negative societal impact. They explain shortcomings in the current framework an point out that AutoML raises CO2 omission.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper elicited significant discussion. The reviewers agreed that the proposed framework could be a valuable tool and could potentially increase the uptake of AutoML systems. The user study, while small, also boosted confidence in the system's utility. The paper is also well-written. 

On the negative side, while the framework is novel in aggregate, the individual components of the framework mostly reflect known ideas. Also (less importantly in my opinion), the paper does not adequately demonstrate that this framework is competitive in performance with state-of-the-art, non-pipeline-based AutoML systems.

In the end, though, the paper's positives outweigh the negatives. After calibrating across my pile, I am recommending acceptance. Please make sure to incorporate the discussion in the review thread in the paper's final version.

Decision: Accept (Poster)