A Framework for Model Search AcrossMultiple Machine Learning Implementations
Yoshiki Takahashi
Tokyo Institute of Technology
Tokyo, Japan

Masato Asahara†
NEC System Platform Research Laboratories
Kanagawa, Japan

Abstract—Several recently devised machine learning (ML)algorithms have shown improved accuracy for various predictive
problems. Model searches, which explore to find an optimal ML
algorithm and hyperparameter values for the target problem,
play a critical role in such improvements. During a model search,data scientists typically use multiple ML implementations toconstruct several predictive models; however, it takes significant
time and effort to employ multiple ML implementations due tothe need to learn how to use them, prepare input data in several
different formats, and compare their outputs. Our proposed
framework addresses these issues by providing simple and unified
coding method. It has been designed with the following two
attractive features: i) new machine learning implementations canbe added easily via common interfaces between the frameworkand ML implementations and ii) it can be scaled to handle large
model configuration search spaces via profile-based scheduling.
The results of our evaluation indicate that, with our framework,
implementers need only write 55-144 lines of code to add a new
ML implementation. They also show that ours was the fastest
framework for the HIGGS dataset, and the second-fastest for the
SECOM dataset.
Keywords-automated machine learning, parallel computing,parameter search, scheduling

I. I NTRODUCTION
Machine learning (ML) techniques are frequently applied
to predictive analytics in a range of industries and academic
fields, such as demand [1], price [2], and click-through rate
[3] prediction. Modern ML implementations, such as XGBoost[4], TensorFlow [5], and scikit-learn [6], enable us to achievea higher accuracies than are possible with traditional ML
algorithms, including linear regression and random forest. The
development of these high-performance ML implementations
has accelerated the application of ML to various types ofpredictive analytics.
As the no-free-lunch theorem shows [7], no single ML
algorithm can achieve the lowest possible loss value for all
loss functions, suggesting that we need to search for a different
optimal ML algorithm for each prediction problem. Data
scientists usually try several different ML algorithms using
multiple ML implementations, because each implementation
supports several algorithms. However, these types of model
searches require a significant amount of time and effort on the† Current affiliation as of August 2019: dotData, Inc.

Kazuyuki Shudo
Tokyo Institute of Technology
Tokyo, Japan

scientists’ part, implementations, prepare the data in suitableformats, and compare their outputs.
We propose a distributed framework for conducting model
searches across multiple ML implementations. Our frameworkhas two particularly attractive features. First, we can plug ina variety of ML implementations, even if they require data in
different formats or programs written in different languages.
To achieve this, we employ common data formats and interfaces that conceal the differences between the framework and
ML implementations. Second, the framework can be scaled via
parallelism. We propose a profile-based scheduling approachthat profiles the processing times of several training tasks anduses that information to schedule the tasks and assign them toworkers.
Our proposed framework enables implementers to add a
new ML implementation with only 55-144 lines of code,significantly less than would be required to implement such
ML algorithms directly. The results of our model searchevaluation show that our framework was the fastest for the
HIGGS dataset, and second-fastest for the SECOM dataset.
This paper is an extended version of our previous work [8],[9] limited to two pages, thus, could only give a brief description of the software’s features and present some preliminary
performance evaluation results. Here, we use different datasets
and frameworks for the comparison, yielding completely different results.
The remainder of the paper is organized as follows. First,
we review the background to model search and discuss our
objectives in Section II. Then we describe our framework’s
design in Section III, before discussing related work in Section
IV. Next, we present and evaluate our experimental results in
Section V. Finally, Section VI concludes the paper.
II. M ODEL S EARCH AND P ROPOSED F RAMEWORKOur aim is to obtain models that can accurately predictlabels or target values for unseen future data. Such classification or regression problems can be handled by several
ML techniques. To find the most accurate predictive model
for a given problem and dataset, we have to search for an
optimal model, i.e., an optimal algorithm with the best possible
hyperparameter values.
Usually, data scientists train their models using pre-existing
ML implementations, such as XGBoost [4], TensorFlow [5], orProc. 15th Int'l eScience Conference (eScience 2019),
pp.381-388, September 2019

scikit-learn [6], instead of implementing the algorithm themselves. Although scikit-learn [6] and Spark MLlib [10] provide
multiple ML algorithms, these do not perform as well as later
implementations (e.g., XGBoost) for some ML algorithms. It
is therefore important to use multiple ML implementations
during the model search process.
That said, however, searching for a model across multiple
ML implementations requires considerable effort on the partof data scientists. First, they must learn how to use each
implementation and read through the documentation for all its
modules to code a model search program for each one. Second,
they must prepare the data in several formats, as different implementations require their own specific formats, such as roworiented, column-oriented, or sparse-matrix formats. Finally,
they have to write a program to compare all the output models
after training has been completed and select the best one.
Therefore, we propose a distributed framework that cansearch for a predictive model across multiple ML implementations. This enables data scientists to execute and compare
a wide range of different algorithm implementations using asingle data format and a unified coding method for describing
the search space. For example, they could use it to conduct
model searches by running XGBoost and TensorFlow implementations in parallel, without having to consider their specificdata formats and coding methods.
Our framework addresses two important challenges. The
first is designing the software so that new ML implementations
can easily be added to the framework, while the other involves
scheduling the training tasks, assigning them to execution units
so that they all require approximately the same processingtime.
A. Simple and Unified Coding Method
Our framework enables data scientists to use a single data
format and a unified coding method to conduct model searches
across multiple implementations, as shown in Figure 1. Here,the first half of the code specifies the search space (possible
ML implementations and hyperparameter values), while thesecond half performs the model search. In this example, we
explore 27 XGBoost configurations, 12 for TensorFlow, and 5
for scikit-learn. Then, the second half of the program reads thetraining and validation data in a unified format and inputs themto the model search functions within the given search space.
This example shows how we can specify a model search in
only 31 lines of code.
As well as simplifying the description of model search, we
can also plug in a variety of ML implementations. Although
our framework is written in Scala, it can also incorporate
implementations written in other languages. For example, ourevaluation (Section V) involved Python libraries for TensorFlow and scikit-learn.
B. Adding New Implementations
Our framework can easily be extended to incorporate new
ML implementations. Recent years have seen the rapid development of new ML algorithms, yielding improved accuracy

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

/****** Search Space ********/
val xgbGrid = new GridBuilder().addGrid(XGBoostParam.eta, Array(0.1,0.3,0.9)).addGrid(XGBoostParam.round, Array(30, 60, 90))
.addGrid(XGBoostParam.maxBin, Array(32,64,128)).buildval tfGrid = new GridBuilder().addGrid(TensorFlowParam.network,
Array("128_128","64_64","128_64","64_64_64")).addGrid(TensorFlowParam.learningRate,
Array(0.003, 0.03, 0.3)).buildval sklearnLRGrid = new GridBuilder().addGrid(ScikitLearnParam.algorithm,
Array("logistic_regression")).addGrid(ScikitLearnParam.c,Array(0.011, 0.033, 0.1, 0.3, 0.9)).build

19
20
21
22
23
24
25
26
27
28
29
30
31

/****** Model Search *******/
val trainDF = spark.read.load("../path/to/data")
val validateDF = spark.read.load("../path/to/data")
val searcher = new ModelSearcher().addSpace(xgbGrid).addSpace(tfGrid).addSpace(sklearnLRGrid).setFeaturesCol("features").setLabelCol("label")val multiModel = searcher.modelSearch(trainDF)
val scores = multiModel.validateAll(validateDF,"features", "label")
Fig. 1.

Sample Scala code for a model search using our framework.

for various prediction problems. Thus, if we want to produce accurate predictions, we need to search for an optimalmodel within a space including these new algorithms. Insteadof implementing them ourselves, we should use third-party
implementations for training.
However, there are several challenges involved in designing
software that can flexibly incorporate multiple ML implementations. One problem is the different formats used by each
ML implementation to store the data. These can include roworiented, column-oriented, and sparse-matrix formats, using
32- or 64-bit values. Thus, our framework must be able
to handle multiple data structures. Another problem is thatinterfacing with each implementation requires us to write a
program in a specific language. To scale our framework tohandle multiple implementations, we overcome these problems
by common interfaces described in Section III-B.
C. Scaling up to Handle Large Search Spaces
The space of model configurations covered during a model
search can be very large, including various combinations of
ML algorithms and hyperparameter values. These have to be
adjusted carefully, because the prediction accuracy can be very
sensitive to the values used. Since no one predictive model
can be optimal for all predictive problems (by the no-freelunch theorem), we need to search through several different
algorithms. Additionally, users may not evaluate predictivemodels based on accuracy alone, but also consider various
other aspects, such as the execution time or the model’stransparency, depending on the application.

Proc. 15th Int'l eScience Conference (eScience 2019),
pp.381-388, September 2019In general, complex learning models, such as deep learning,are likely to achieve high accuracy, but if we focus more ontransparency then simpler models, such as linear ones, may be
preferable. Thus, in order to find an optimal model based on
several different evaluation metrics, it is often better to explore
a variety of algorithms, from simple to more complex models.
Even with sophisticated parameter tuning techniques, such as
Bayesian optimization [11], it can take a very long time to
explore such search spaces.
The time required to search through a large space can be
reduced by parallel computing. Even though a single machine
may take several days, or even several tens of days, to complete a search, we can accelerate this through parallelization,for example by dividing the search space into equal parts andassigning a separate machine to explore each part. That said,
it is difficult to reduce the processing time optimally simply
by increasing the number of machines: since the training
time can vary widely depending on the learning algorithm
and hyperparameters used, the time taken by each machine
to finish its exploration process can vary widely. Thus, theslowest processing node dictates the total time required forthe search and the performance improvement may not be ideal.
Section III-C describes our proposed profile-based schedulingsolution.
III. F RAMEWORK D ESIGNIn designing our framework, our aim was to make it easy
to incorporate multiple ML implementations and thereby findoptimal predictive models within large search spaces. This
raised two issues. The first is the different data formats andinterfaces used by the implementations, making it difficult toinclude them in one unified framework. The second issue isthe difficulty of scheduling the model search in parallel acrossmultiple machines to achieve good performance scaling.
In this section, we discuss the framework’s overall architecture and the techniques used to address these issues. It
comprises four main modules, which we describe in Section
3.1. Then, we show how the software is designed to makeadding new ML implementations easy (Section III-B), andhow we use profile-based scheduling to reduce the overall
execution time (Section III-C).
A. Architecture
Our framework comprises four main modules: Driver, Hyperparameter Tuner, Scheduler and several Executors (Fig. 2).
Here, we discuss their functions during a model search.
First, the user provides a dataset, an evaluation metric (forvalidation), and a suggested search space to the Driver. Then,the Driver passes the search space to the Hyperparameter
Tuner, which returns a set of training configurations. The
Hyperparameter Tuner uses a static hyperparameter tuningalgorithm, such as grid or random search [12], to generate
a set of model configurations. Then, the Driver queries the
Scheduler as to which Executor should execute which subset
of configurations. The Scheduler balances the load among
the Executors by assigning them tasks based on profilingFig. 2.

Architecture of our framework.

information (Section III-C). The Driver then runs training
tasks on all the Executors using the Apache Spark [13] map
function and obtains the trained predictive models. Thesemodels are then evaluated by the Executors, similarly to thetraining process, and the best model (according to the given
evaluation metric) is returned to the user.
Incorporating a new ML implementation into the framework
involves making it runnable by the Executors. In Section III-B,
we discuss how the framework is designed to be extended,enabling ML implementations using different languages ordata structures to work correctly on the Executors. In Section
III-C, we introduce the method adopted by the Scheduler toscale up the search to large search spaces.
B. Common Interface to Hide Implementation Differences
Every time a new learning algorithm is devised, the model
search space must be extended. Compared with implementing
a new ML algorithm from scratch, simply writing glue code
to reuse an existing algorithm implementation can drasticallyreduce the amount of additional code required. However,plugging in a new implementation in this way requires the
Driver to be modified to handle the differences between theimplementations.
To simplify this, we have designed common interfaces,
which the Driver uses to call training and prediction functions,which conceal the interface differences. Since the Driveralways uses the same interfaces to call these functions,
implementers do not need to make any changes to it inorder to plug in new implementations. Instead, they simply
define new training and prediction functions that inherit fromthese interfaces and invoke new third-party functions withinthem. Additionally, there is no need to consider distributed
processing because the training processes are executed locallyon the Executors.
The gap of data formats is resolved on the Executors. The
common interface discussed above takes uniformed format

Proc. 15th Int'l eScience Conference (eScience 2019),
pp.381-388, September 2019

HIGGS

6

SECOM
15.0
12.5

Processing time (min)Model search
Profiling8
Processing time (hour)

data, which is represented as a row-oriented dense matrix.
This uniformed format data is going to be converted into a
specific format for new implementation just before training.
The data format differences are resolved by the Executors.
The common interface discussed above takes data in a uniform
format, represented as a row-oriented dense matrix. This isthen converted into the format needed by each implementation
immediately prior to training.
Ideally, the common format should be determined basedon the target data. Tables containing mostly zero or empty
values can be more heavily compressed by storing them in asparse-matrix format instead of a dense format, while thosecontaining few zero values will be poorly compressed by asparse-matrix format. Our framework does not support thevariety of these formats yet but a dense matrix representation
as a common format.

10.0

4
2

2.3 %

0
Fig. 3.

7.5
5.0
2.5
0.0

7.2 %

Ratio of profiling time to total processing time.

C. Profile-based Scheduling
It is difficult to optimally reduce the processing time byincreasing the number of Executors. For example, dividing thesearch space into equal parts and assigning them to different
Executors will not generally improve performance as much as
expected, as different machines will have different processing
times (Section II-C).
Here, we propose a scheduling method that uses profiling
results to balance the task loads. The overall processing time
can be expressed by the maximum processing time over
all nodes. Assuming that all the task processing times are
known, allocating the training tasks to nodes is an instance ofan optimization problem known as job-shop-scheduling [14].
Since this problem is NP-hard [15], we solve it using anapproximate (greedy) method.
To solve job-shop-scheduling, the framework must know all
the task processing times. First, it samples a few percent of thedata from the whole dataset. This is used for training, to profile
the processing times of several training tasks in the search
space. Based on assuming that the training time is proportional
to the data size, the Scheduler uses the profiling time divided
by the sampling rate as the task processing time.
This scheduling method is useful when the time requiredfor profiling is sufficiently short compared with the overall
execution time. Figure 3 shows the ratio of the profiling time tothe total processing time for the HIGGS and SECOM datasets(Section V-A) based on exploring four ML algorithms and1,211 model variants. We sampled 1% and 3% of the data,
respectively, to profile the datasets and solved the job shop
scheduling problem using a greedy method. The other settings
were as as described in Section V-A. Here, profiling required
less than 8% of the overall execution time, which we believe
is sufficiently fast.
Although we only discuss static scheduling here, it isalso necessary to predict the processing time for dynamicscheduling. In dynamic scheduling, once a worker has finished
one job, it acquires the next one, and all nodes continue
to operate until no unexecuted jobs remain. Even with such
dynamic scheduling, if the last job assigned to a worker is along one, the other workers may have to wait for that worker tocomplete it, meaning we need to account for this in predicting
the processing time.
IV. R ELATED WORKIn this section, we discuss two topics related to model
search: distributed model search frameworks, and hyperparameter tuning techniques.
A. Distributed Model Search Frameworks
There are several distributed frameworks that are designed
to accelerate the time-consuming predictive model construction. The Spark MLlib [10] framework aims to scale up model
search performance to huge amounts of data. It provides several ML algorithms and a function to automatically search for
optimal predictive models based on hyperparameter values foreach learning algorithm. By default, model searches evaluate
the model configurations serially, but it can be configured tooperate in parallel. However, since it trains a single modelin parallel, the communication overhead caused by its shuffle
process affects performance significantly more than with ourframework. Furthermore, it is difficult to extend existing ML
implementations to run in parallel for use with MLlib.
The spark-sklearn [16] framework runs scikit-learn’s model
search method on multi-node clusters with Apache Spark. It
can explore all ML algorithms supported by scikit-learn, andapply grid-search hyperparameter optimization. Its scheduler
divides training tasks into groups and assigns one group toeach worker. Our framework differs from spark-sklearn inthat it can incorporate multiple ML implementations and use
profile-based scheduling.
The MLbase [17] framework is designed to construct predictive models using distributed systems based on simple
descriptions. It enables users to execute complicated steps,
such as preprocessing, model verification, and model selection,
without deep system knowledge. TuPAQ [18] , building on
MLbase, can obtain highly accurate predictive models in ashort time using a scheduling method called bandit resource
allocation that assigns computing resources to regions of theProc. 15th Int'l eScience Conference (eScience 2019),
pp.381-388, September 2019search space where more accurate models are likely to lie.
By contrast, our framework’s scheduling approach aims tominimize the overall execution time required to complete all
the training tasks.
The HyperDrive [19] framework’s scheduling method has
evolved over time into bandit resource allocation. However,bandit resource allocation cannot cope with cases where theaccuracies of two models reverse their order after several
learning iterations. To deal with this problem, HyperDrive’sscheduling approach is based on the potential future prediction
accuracy.
B. Hyperparameter Tuning Algorithms
Several proposed algorithms aim to construct more accurate
predictive models by selecting which model configurations
to explore within the search space. Grid search and random
search [12] are two classic hyperparameter tuning algorithms.
Grid search tries all the grid points between hyperparametervalues in a pre-determined set. Since the user can manually
determine the hyperparameter values that are actually considered, this approach is suitable for analyzing the impact ofparticular hyperparameters on accuracy. The random searchmethod evaluates several sets of randomly selected values fromthe search ranges specified for each hyperparameter.
In contrast to these static methods, there are also several
techniques for dynamically determining future hyperparametervalues to consider based on previously evaluated configurations [20], [11], [21], [22]. Bayesian optimization [20], [11]is frequently used for this, and has been adopted by several
frameworks, such as Auto-WEKA [23] and Hyperopt [24].
Our framework can utilize these algorithms by incorporatingthem into its Hyperparameter Tuner. This generates model
configurations to evaluate based on a user-specified tuning algorithm. When using dynamic tuning algorithms, it iteratively
receives the evaluation results and uses them to generate newmodel configurations.
V. EVALUATION
Now that we have discussed the two issues our frameworkaddresses, and their solutions, we investigate its performanceexperimentally. First, to demonstrate that the framework caneasily incorporate new ML implementations, we measure thechanges in the number of lines of code after adding several third-party implementations. Second, we perform model
searches using the framework to compare its performance withthose of existing frameworks.
A. Experimental Setupa) Workloads: We used two binary classification datasetswith numeric-valued attributes, taken from a UCI dataset
repository [25]. The HIGGS dataset1 is derived from Monte
Carlo simulations of physics events. From this, we sampled
100,000 instances each for the training, validation, and testing
sets and standardized them for use in model search. The
SECOM dataset2 consists of signal data collected from sensors
1 https://archive.ics.uci.edu/ml/datasets/HIGGS
2 https://archive.ics.uci.edu/ml/datasets/secomand process measurement points used in semiconductor manufacturing. This included 1,567 instances, which we divided
in a 6:2:2 ratio into training, validation, and testing sets, thenstandardized.
Using our framework, we executed model searches with 1,
2, 4, 8, 16, and 32 parallel tasks, measuring their execution
times. All the experiments were conducted on x86-64 servers,
each with a single 4-core Intel Xeon E3-1280 CPU running at
3.70 GHz and 32 GB of memory. We used one such machinefor 1, 2, and 4 parallel tasks, and clusters of 2, 4, and 8
machines for 8, 16, and 32 tasks, respectively.
Here, we compared the results for two scheduling methods:
random scheduling, which randomly assigns equal numbersof training tasks to all nodes, and the proposed profile-basedscheduling method. For profiling, we sampled 1% and 3% of
the data for the HIGGS and SECOM datasets, respectively,
and used the results to train models for all configurations inthe search space. The scheduler then solved the optimization
problem using a greedy algorithm, estimating the processing
time by dividing the training times by the sampling rate.
b) Model search space: We considered four algorithms,
exploring a total of 1,211 training configurations (algorithmhyperparameter pairs) using grid search. Gradient boosting tree
is an ensemble learning algorithm that combines many decision trees. For this algorithm, we explored 864 configurations
by changing 6 hyperparameters, and executed it by running
XGBoost [4] inside the framework. Multilayer perceptrons(MLPs) are artificial neural networks where each layer weights
its input values and then applies nonlinear functions to produce
output that it sends to the following layer. Here, we used
TensorFlow [5] to train the models using 324 configurations,changing factors such as the numbers of layers and neurons
and the learning rate. Finally, we also considered random
forests and logistic regression, two traditional ML algorithms,
which we trained using scikit-learn [6], using 18 and 5
configurations respectively.
B. Comparison with Existing Frameworks
We compared our framework with two existing frameworks:
Spark MLlib and spark-sklearn. Using these frameworks,
we ran model searches with the same ML algorithms andhyperparameter values described in Section V-A. Spark MLlib[10] is a machine learning library, aimed at distributed largescale data, which provides hyperparameter tuning and model
selection functions, as well as several ML algorithms. The
spark-sklearn [16] framework runs scikit-learn’s model searchon multi-node clusters with Apache Spark. We compared thesewith our framework in terms of the execution time and theaccuracy for model search.
Spark MLlib also has a parameter that enables several models to be evaluated in parallel during model search, although
it evaluates the models serially by default. In this evaluation,
we investigated its performance for parameter values of 4-12
and selected 8 for that parameter.

Proc. 15th Int'l eScience Conference (eScience 2019),
pp.381-388, September 2019

scikit-learn (The part of ML algorithm)
Our framework (Scala part)Our framework (Python part)

1200
928

905

800

680

629

600
400
200
0

55

131

144

142

ron
rest
tree
sion
cep)t
grersn) ndom foarn)rtingost)
s
e
e
r
o
p
o
c
r
i
w
le
B XGBoilaye rFlo Logisstcikit-lea Ra(scikit(
Mult (Tenso(

Fig. 4. Numbers of code lines required to incorporate different ML algorithms
into our framework.
TABLE II MPLEMENTATIONS USED TO INCORPORATE EACH ALGORITHM INTO OURFRAMEWORK .

Boosting tree
Multilayer perceptron
Logistic regression
Random forest

Our framework(XGB, TF, sklearn)
XGBoostTensorFlow
scikit-learn
scikit-learn

HIGGSProfile-based scheduling
Random scheduling
Ideal scaling

SpeedupLines of code

1000

32
28
24
20
16
12
8
4
2

Our framework(sklearn)scikit-learn
scikit-learn
scikit-learn
scikit-learn

32
28
24
20
16
12
8
4
2

64.2%

64.6%

37.2%

46.3%

55.4%
8

16
Cores

32

SECOM

Fig. 5.

Profile-based scheduling
Random scheduling

65.5%

Ideal scaling

77.2%
69.4%
72.2%
12 4

47.2%

79.6%

64.1%

69.3%
8

16
Cores

32

Improvements in performance due to increased parallelism.

C. Results
Figure 4 shows the number of lines of code required to
add each new algorithm to our framework, with the red andblue bars showing the numbers of lines of Scala and Python
code required, respectively. This shows that we can include
a new ML implementation with only 55-144 lines of code.
Although ML implementations that support the Java API (e.g.,
XGBoost) can easily be added to our framework because itis written in Scala, Python libraries, such as TensorFlow andscikit-learn, are more costly to incorporate. Even so, comparedwith implementing each ML algorithm directly (gray bars),
only needing to write glue code for our framework reduced
the implementation cost drastically.
To evaluate our scheduling performance, we compared
our proposed profile-based scheduling method with random
scheduling for different numbers of cores. Figure 5 plots
the performance improvements as a percentage of the ideal
scaling, based on the results for one core. Compared withrandom scheduling, the performance of our profile-basedscheduling approach was significantly better when the degreeof parallelism was large. Even though our method requires
some profiling time at the start of the search process, it canstill improve performance considerably.
Figure 6 compares the total model search execution time forour framework with those for Spark MLlib and spark-sklearn.
Here, the colored bars indicate the results for our framework
when it incorporated both XGBoost (XGB), TensorFlow (TF),
and scikit-learn (sklearn), as shown in Table I (blue bars),
and with only scikit-learn (green bars). These results show

64.1%

70.0%

63.9%

12 4

Speedup

1400

Spark MLlib

HIGGS
87.2

3.7

spark-sklearn
Our framework(XGB, TF, sklearn)
Our framework(sklearn)

3.5
6.9
0

20

Spark MLlib

SECOM
550.6

spark-sklearn

6.3

Our framework(XGB, TF, sklearn)

80

200
300
400
Execution time (min)

500

15.6
79.2

Our framework(sklearn)

40
60
Execution time (hour)

0100Fig. 6. Comparison of model search execution times for our framework withthose for Spark MLlib and spark-sklearn.

that combining all three ML implementations gave better
performance than using scikit-learn alone, demonstrating that
our framework can benefit from newer implementations withhigher performance than traditional ones.
However, comparing our framework (sklearn) with sparksklearn shows that spark-sklearn achieved better performance,even though they both ran scikit-learn during training. Afterinvestigating this, we found that the performance difference
was due to the overhead of generating new Python processes

Proc. 15th Int'l eScience Conference (eScience 2019),
pp.381-388, September 2019

HIGGS
Spark MLlib

0.801

spark-sklearn
Our framework(XGB,TF,sklearn)
Our framework

0.818
0.806
0.811

)
Worst result *(sklearn

0.674

0.0

0.2

0.4
0.6
Achieved highest AUC

0.8

1.0

SECOM
Spark MLlib

0.66

spark-sklearn
Our framework(XGB,TF,sklearn)
Our framework

0.684
0.674

)
Worst result **0.685(sklearn

0.0

0.614

0.2

0.4
0.6
Achieved highest AUC

0.8

1.0

* Searched for multilayer perceptron on MLlib** Searched for multilayer perceptron on TensorFlow (Our framework)

Fig. 7. Achieved AUC accuracy for the three frameworks: Spark MLlib,
spark-sklearn and our framework. ”Worst result” shows the case of searching
only one ML algorithm for the model search.

and passing training data to them via the file system. Thus,
there is room for performance improvement if we can either
reuse the same Python process or utilize a serialization format
with low serialization and deserialization costs. Even so, despite suffering the same overhead when combining XGB, TF,
and sklearn, our framework was still the fastest at conducting
a model search for the HIGGS dataset.
Spark MLlib was slower than the other frameworks due tothe overhead imposed by the shuffle process needed to train
a single model in a distributed fashion. MLlib distributes thetraining data among the nodes and manages it, constructing
one model by training local models and communicating thembetween nodes in a shuffle process.
Figure 7 shows the area under the ROC curve (AUC)
accuracies achieved by all the frameworks. In addition to theresults quoted previously, this also shows the worst results
achieved in each experiment, which were for investigating
the MLP algorithm, using MLlib for the HIGGS dataset andTensorFlow for the SECOM dataset. Comparing the otherresults with the worst ones shows that all the frameworks
achieved better accuracies by considering multiple algorithms
than with only a single algorithm.
Figure 7 also confirms that all the frameworks achieved
almost the same accuracy, and thus executed model search correctly. Since all four results involved the same set of algorithms
and our framework depends on external implementations, it issufficient to ensure that all the frameworks achieved almost
the same accuracy.
VI. C ONCLUSION
In this paper, we have proposed a distributed frameworkfor performing model search across multiple ML implementa-

tions. Our framework has two attractive features: i) new ML
implementations can be added by writing common interfaces
between the framework and the ML implementations and ii)
it scales well with the degree of parallelism due to its profilebased scheduling approach.
We have also shown that, with our framework, implementersneed only write 55-144 lines of code to add a new ML
implementation. Additionally, our framework achieved model
search results that were the fastest for the HIGGS dataset, andsecond-fastest for the SECOM dataset.
In future work, we plan to reduce the overhead causedby adding ML implementations in different languages toour framework. The process invocation overhead could be
improved by reusing the same process once called, and theoverhead due to passing training data via files could be
improved by using a format, such as Apache Arrow, that has
low serialization and deserialization costs.
Another problem worth investigating is to assess how theoverall execution time is affected by the data sampling rateused for profiling and the number times the profiling stepis executed. Increasing either of these would increase theoverall execution time due to the profiling time required, but
reducing either of them significantly could adversely affectscheduling performance. Thus, we also plan to study the
profiling parameters needed to minimize the overall execution
time in future work.
ACKNOWLEDGMENT

This work was supported by New Energy and Industrial
Technology Development Organization (NEDO).
R EFERENCES[1] J. Parra and C. Kiekintveld, Initial exploration of machine learning topredict customer demand in an energy market simulation. AI Access
Foundation, 1 2013, vol. WS-13-15, pp. 29–32.
[2] H. Ince and T. B. Trafalis, “Kernel principal component analysis andsupport vector machines for stock price prediction,” IIE Transactions,vol. 39, no. 6, pp. 629–637, 2007.
[3] H. B. McMahan, G. Holt, D. Sculley, M. Young, D. Ebner, J. Grady,
L. Nie, T. Phillips, E. Davydov, D. Golovin, S. Chikkerur, D. Liu,
M. Wattenberg, A. M. Hrafnkelsson, T. Boulos, and J. Kubica, “Ad click
prediction: A view from the trenches,” in Proceedings of the 19th ACM
SIGKDD International Conference on Knowledge Discovery and Data
Mining, ser. KDD ’13. New York, NY, USA: ACM, 2013, pp. 1222–
1230. [Online]. Available: http://doi.acm.org/10.1145/2487575.2488200[4] T. Chen and C. Guestrin, “XGBoost: A scalable tree boosting system,”in Proceedings of the 22nd ACM SIGKDD international conference onknowledge discovery and data mining. ACM, 2016, pp. 785–794.
[5] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S.
Corrado, A. Davis, J. Dean, M. Devin et al., “Tensorflow: Large-scalemachine learning on heterogeneous distributed systems,” arXiv preprint
arXiv:1603.04467, 2016.
[6] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion,
O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg et al.,“Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12 Oct, pp. 2825–2830, 2011.
[7] D. H. Wolpert, W. G. Macready et al., “No free lunch theorems forsearch,” Technical Report SFI-TR-95-02-010, Santa Fe Institute, Tech.
Rep., 1995.
[8] Y. Takahashi, M. Asahara, and K. Shudo, “A framework for searchinga predictive model,” SysML Conference 2018, 2018.
[9] M. Asahara, Y. Takahashi, and K. Shudo, “Quick! Quick! Exploration!:
A framework for searching a predictive model on Apache Spark,” in
DataWorks Summit San Jose 2018, 2018.

Proc. 15th Int'l eScience Conference (eScience 2019),
pp.381-388, September 2019[10] X. Meng, J. Bradley, B. Yavuz, E. Sparks, S. Venkataraman, D. Liu,
J. Freeman, D. Tsai, M. Amde, S. Owen et al., “MLlib: Machine learning
in apache spark,” The Journal of Machine Learning Research, vol. 17,
no. 1, pp. 1235–1241, 2016.
[11] J. Snoek, H. Larochelle, and R. P. Adams, “Practical bayesian optimization of machine learning algorithms,” in Advances in neural information
processing systems, 2012, pp. 2951–2959.
[12] J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization,” Journal of Machine Learning Research, vol. 13 Feb, pp.
281–305, 2012.
[13] M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica,“Spark: Cluster computing with working sets,” in Proceedings ofthe 2nd USENIX Conference on Hot Topics in Cloud Computing,ser. HotCloud’10. Berkeley, CA, USA: USENIX Association, 2010.
[Online]. Available: http://dl.acm.org/citation.cfm?id=1863103.1863113[14] M. R. Garey, D. S. Johnson, and R. Sethi, “The complexityof flowshop and jobshop scheduling,” Mathematics of Operations
Research, vol. 1, no. 2, pp. 117–129, 1976. [Online]. Available:
http://www.jstor.org/stable/3689278[15] Y. Sotskov and N. Shakhlevich, “NP-hardness of shop-schedulingproblems with three jobs,” Discrete Applied Mathematics,
vol. 59, no. 3, pp. 237 – 266, 1995. [Online]. Available:
http://www.sciencedirect.com/science/article/pii/0166218X9580004N[16] B. Joseph, H. Tim, and V. Feinberg. (2015) Scikit-learn integration
package for apache spark. https://github.com/databricks/
spark-sklearn (accessed Jan. 10, 2019).
[17] T. Kraska, A. Talwalkar, J. C. Duchi, R. Griffith, M. J. Franklin, and M. I.
Jordan, “MLbase: A distributed machine-learning system,” in CIDR,
vol. 1, 2013.
[18] E. R. Sparks, A. Talwalkar, D. Haas, M. J. Franklin, M. I. Jordan, and
T. Kraska, “Automating model search for large scale machine learning,”
in Proceedings of the Sixth ACM Symposium on Cloud Computing, ser.
SoCC ’15. New York, NY, USA: ACM, 2015, pp. 368–380. [Online].
Available: http://doi.acm.org/10.1145/2806777.2806945[19] J. Rasley, Y. He, F. Yan, O. Ruwase, and R. Fonseca, “HyperDrive:
Exploring hyperparameters with POP scheduling,” in Proceedings of the
18th International Middleware Conference, Middleware, vol. 17, 2017.
[20] D. R. Jones, M. Schonlau, and W. J. Welch, “Efficient global optimization of expensive black-box functions,” Journal of Global optimization,
vol. 13, no. 4, pp. 455–492, 1998.
[21] Y.-Q. Hu, Y. Yu, and Z.-H. Zhou, “Experienced optimization withreusable directional model for hyper-parameter search.” in IJCAI, 2018,
pp. 2276–2282.
[22] B. Bowen, G. Otkrist, N. Nikhil, and R. Ramesh, “Designing neuralnetwork architectures using reinforcement learning,” in Proceedings of
5th International Conference on Learning Representations (ICLR17),2017.
[23] C. Thornton, F. Hutter, H. H. Hoos, and K. Leyton-Brown, “AutoWEKA: Combined selection and hyperparameter optimization ofclassification algorithms,” in Proceedings of the 19th ACM SIGKDDInternational Conference on Knowledge Discovery and Data Mining,ser. KDD ’13. New York, NY, USA: ACM, 2013, pp. 847–855.
[Online]. Available: http://doi.acm.org/10.1145/2487575.2487629[24] J. Bergstra, D. Yamins, and D. D. Cox, “Hyperopt: A Python libraryfor optimizing the hyperparameters of machine learning algorithms,” inProceedings of the 12th Python in Science Conference, 2013, pp. 13–20.
[25] D. Dheeru and E. Karra Taniskidou, “UCI machine learning repository,”
2017. [Online]. Available: http://archive.ics.uci.edu/ml

Proc. 15th Int'l eScience Conference (eScience 2019),
pp.381-388, September 2019