Package: piex
An open source project from Data to AI Lab at MIT.Pipeline ExplorerClasses and functions to explore and reproduce the performance obtained by
thousands of MLBlocks pipelines and templates across hundreds of datasets.Free software: MIT licenseDocumentation:https://HDI-Project.github.io/piexHomepage:https://github.com/HDI-Project/piexOverviewThis repository contains a collection of classes and functions which allows a user to easily
explore the results of a series of experiments run by team MIT using MLBlocks pipelines over
a large collection of Datasets.Along with this library we are releasing a number of fitted pipelines, their performance on
cross validation, test data and metrics. The results of these experiments were stored in a
Database and later on uploaded to Amazon S3, from where they can be downloaded and analyzed
using the Pipeline Explorer.We will continuously add more pipelines, templates and datasets to our experiments and make
them publicly available to the community.These can be used for the following purposes:Find what is the best score we found so far for a given dataset and task type (given the
search space we defined and our tuners)Use information about pipeline performance to do meta learningCurrent summary of our experiments is:# ofdatasets453pipelines2115907templates63tests2152ConceptsBefore diving into the software usage, we briefly explain some concepts and terminology.PrimitivesWe call the smallest computational blocks used in a Machine Learning processprimitives, which:Can be either classes or functions.Have some initialization arguments, which MLBlocks callsinit_params.Have some tunable hyperparameters, which have types and a list or range of valid values.TemplatesPrimitives can be combined to form what we callTemplates, which:Have a list of primitives.Have some initialization arguments, which correspond to the initialization arguments
of their primitives.Have some tunable hyperparameters, which correspond to the tunable hyperparameters
of their primitives.PipelinesTemplates can be used to buildPipelinesby taking and fixing a set of valid
hyperparameters for a Template. Hence, Pipelines:Have a list of primitives, which corresponds to the list of primitives of their template.Have some initialization arguments, which correspond to the initialization arguments
of their template.Have some hyperparameter values, which fall within the ranges of valid tunable
hyperparameters of their template.A pipeline can be fitted and evaluated using the MLPipeline API in MLBlocks.DatasetsA collection of ~450 datasets was used covering 6 different data modalities and 17 task types.Each dataset was split using a holdout method in two parts, training and testing, which were
used respectively to find and fit the optimal pipeline for each dataset, and to later on
evaluate the goodness-of-fit of each pipeline against a specific metric for each dataset.This collection of datasets is stored in an Amazon S3 Bucket in theD3M format,
including the training and testing partitioning, and can be downloaded both using piex or a web browser following this link:https://d3m-data-dai.s3.amazonaws.com/index.htmlWhat is an experiment/test?Throughout our description we will refer to a search process as anexperimentor atest.
An experiment/test is defined as follows:It is given a dataset and a taskIt is given a templateIt then searches using a Bayesian tuning algorithm (using a tuner from our BTB library). Tuning
algorithm tests multiple pipelines derived from the template and tries to find the best set of
hyperparameters possible for that template on each dataset.During the search process, a collection of information is stored in the database and is
available through piex. They are:Cross Validation score obtained over the training partition by each pipeline fitted during
the search process.In parallel, at some points in time the best pipeline already found was validated against
the testing data, and the   obtained score was also stored in the database.Each experiment was given one or more of the following configuration values:Timeout: Maximum time that the search process is allowed to run.Budget: Maximum number of tuning iterations that the search process is allowed to perform.Checkpoints: List of points in time, in seconds, where the best pipeline so far was
scored against the testing data.Pipeline: The name of the template to use to build the pipelines.Tuner Type: The type of tuner to use,gporuniform.Getting StartedInstallationThe simplest and recommended way to install the Pipeline Explorer is using pip:pipinstallpiexAlternatively, you can also clone the repository and install it from sourcesgitclonegit@github.com:HDI-Project/piex.gitcdpiex
pipinstall-e.UsageThe S3PipelineExplorerTheS3PipelineExplorerclass provides methods to download the results from previous
tests executions from S3, see which pipelines obtained the best scores and load them
as a dictionary, ready to be used by an MLPipeline.To start working with it, it needs to be given the name of the S3 Bucket from which
the data will be downloaded.For this examples, we will be using theml-pipelines-2018bucket, where the results
of the experiments run for the Machine Learning Bazaar paper can be found.frompiex.explorerimportS3PipelineExplorerpiex=S3PipelineExplorer('ml-pipelines-2018')The DatasetsTheget_datasetsmethod returns apandas.DataFramewith information about the
available datasets, their data modalities, task types and task subtypes.datasets=piex.get_datasets()datasets.shape(453, 4)datasets.head()datasetdata_modalitytask_typetask_subtype314124_120_mnistimageclassificationmulti_class315124_138_cifar100imageclassificationmulti_class316124_153_svhn_croppedimageclassificationmulti_class317124_174_cifar10imageclassificationmulti_class318124_178_coil100imageclassificationmulti_classdatasets=piex.get_datasets(data_modality='multi_table',task_type='regression')datasets.head()datasetdata_modalitytask_typetask_subtype311uu2_gp_hyperparameter_estimationmulti_tableregressionmultivariate312uu3_world_development_indicatorsmulti_tableregressionunivariateThe ExperimentsThe list of tests that have been executed can be obtained with the methodget_tests.This method returns apandas.DataFramethat contains a row for each experiment that has been run on each dataset.
This dataset includes information about the dataset, the configuration used for the experiment, such as the
template, the checkpoints or the budget, and information about the execution, such as the timestamp, the exact
software version, the host that executed the test and whether there was an error or not.Just like theget_datasets, any keyword arguments will be used to filter the results.importpandasaspdtests=piex.get_tests()tests.head().T01234budgetNaNNaNNaNNaNNaNcheckpoints[900, 1800, 3600, 7200][900, 1800, 3600, 7200][900, 1800, 3600, 7200][900, 1800, 3600, 7200][900, 1800, 3600, 7200]commit4c7c29f4c7c29f4c7c29f4c7c29f4c7c29fdataset196_autoMpg26_radon_seedLL0_1027_eslLL0_1028_swdLL0_1030_eradockerFalseFalseFalseFalseFalseerrorNaNNaNNaNNaNNaNhostnameec2-52-14-97-167.us-east-2.compute.amazonaws.comec2-18-223-109-53.us-east-2.compute.amazonaws.comec2-18-217-79-23.us-east-2.compute.amazonaws.comec2-18-217-239-54.us-east-2.compute.amazonaws.comec2-18-225-32-252.us-east-2.compute.amazonaws.comimageNaNNaNNaNNaNNaNinsert_ts2018-10-24 20:05:01.8722018-10-24 20:05:02.7782018-10-24 20:05:02.8792018-10-24 20:05:02.9802018-10-24 20:05:03.081pipelinecategorical_encoder/imputer/standard_scaler/xg...categorical_encoder/imputer/standard_scaler/xg...categorical_encoder/imputer/standard_scaler/xg...categorical_encoder/imputer/standard_scaler/xg...categorical_encoder/imputer/standard_scaler/xg...statusdonedonedonedonedonetest_id2018102420050187208320181024200501872083201810242005018720832018102420050187208320181024200501872083timeoutNaNNaNNaNNaNNaNtracebackNaNNaNNaNNaNNaNtuner_typeNaNNaNNaNNaNNaNupdate_ts2018-10-24 22:05:55.3862018-10-24 22:05:57.5082018-10-24 22:05:56.3372018-10-24 22:05:56.1122018-10-24 22:05:56.164data_modalitysingle_tablesingle_tablesingle_tablesingle_tablesingle_tabletask_typeregressionregressionregressionregressionregressiontask_subtypeunivariateunivariateunivariateunivariateunivariatemetricmeanSquaredErrorrootMeanSquaredErrormeanSquaredErrormeanSquaredErrormeanSquaredErrordataset_id196_autoMpg_dataset_TRAIN26_radon_seed_dataset_TRAINLL0_1027_esl_dataset_TRAINLL0_1028_swd_dataset_TRAINLL0_1030_era_dataset_TRAINproblem_id196_autoMpg_problem_TRAIN26_radon_seed_problem_TRAINLL0_1027_esl_problem_TRAINLL0_1028_swd_problem_TRAINLL0_1030_era_problem_TRAINtargetclasslog_radonout1Out1out1size24160165232size_human24K160K16K52K32Ktest_features7284104test_samples100183100199199train_features7284104train_samples298736388801801pd.DataFrame(tests.groupby(['data_modality','task_type']).size(),columns=['count'])countdata_modalitytask_typegraphcommunity_detection5graph_matching18link_prediction2vertex_nomination2imageclassification57regression1multi_tableclassification1regression1single_tableclassification1405collaborative_filtering1regression430time_series_forecasting175textclassification17timeseriesclassification37tests=piex.get_tests(data_modality='graph',task_type='link_prediction')tests[['dataset','pipeline','checkpoints','test_id']]datasetpipelinecheckpointstest_id171659_umlsNaN[900, 1800, 3600, 7200]20181031040541366347214159_umlsgraph/link_prediction/random_forest_classifier[900, 1800, 3600, 7200]20181031182305995728The Experiment ResultsThe results of the experiments can be seen using theget_experiment_resultsmethod.These results include both the cross validation score obtained by the pipeline during
the tuning, as well as the score obtained by this pipeline once it has been fitted
using the training data and then used to make predictions over the test data.Just like theget_datasets, any keyword arguments will be used to filter the results,
including thetest_id.results=piex.get_test_results(test_id='20181031182305995728')results[['test_id','pipeline','score','cv_score','elapsed','iterations']]test_idpipelinescorecv_scoreelapsediterations746420181031182305995728graph/link_prediction/random_forest_classifier0.4998530.843175900.255511435.0746520181031182305995728graph/link_prediction/random_forest_classifier0.4998530.8546031800.885417805.0746620181031182305995728graph/link_prediction/random_forest_classifier0.4998530.8546033600.0050721432.0746720181031182305995728graph/link_prediction/random_forest_classifier0.7855680.8600007200.2252562366.0The Best PipelineInformation about the best pipeline for a dataset can be obtained using theget_best_pipelinemethod.This method returns apandas.Seriesobject with information about the pipeline that obtained the
best cross validation score during the tuning, as well as the template that was used to build it.Note: This call will download some data in the background the first time that it is run,
so it might take a while to return.piex.get_best_pipeline('185_baseball')id                            17385666-31da-4b6e-ab7f-8ac7080a4d55
dataset                                 185_baseball_dataset_TRAIN
metric                                                     f1Macro
name             categorical_encoder/imputer/standard_scaler/xg...
rank                                                      0.307887
score                                                     0.692113
template                                  5bd0ce5249e71569e8bf8003
test_id                                       20181024234726559170
pipeline         categorical_encoder/imputer/standard_scaler/xg...
data_modality                                         single_table
task_type                                           classification
Name: 1149699, dtype: objectApart from obtaining this information, we can use theload_best_pipelinemethod
to load its JSON specification, ready to be using in anmlblocks.MLPipelineobject.pipeline=piex.load_best_pipeline('185_baseball')pipeline['primitives']['mlprimitives.feature_extraction.CategoricalEncoder',
 'sklearn.preprocessing.Imputer',
 'sklearn.preprocessing.StandardScaler',
 'mlprimitives.preprocessing.ClassEncoder',
 'xgboost.XGBClassifier',
 'mlprimitives.preprocessing.ClassDecoder']The Best TemplateJust like the best pipeline, the best template for a given dataset can be obtained using
theget_best_templatemethod.This returns just the name of the template that was used to build the best pipeline.template_name=piex.get_best_template('185_baseball')template_name'categorical_encoder/imputer/standard_scaler/xgbclassifier'This can be later on used to explore the template, obtaining its default hyperparameters:defaults=piex.get_default_hyperparameters(template_name)defaults{'mlprimitives.feature_extraction.CategoricalEncoder#1': {'copy': True,
  'features': 'auto',
  'max_labels': 0},
 'sklearn.preprocessing.Imputer#1': {'missing_values': 'NaN',
  'axis': 0,
  'copy': True,
  'strategy': 'mean'},
 'sklearn.preprocessing.StandardScaler#1': {'with_mean': True,
  'with_std': True},
 'mlprimitives.preprocessing.ClassEncoder#1': {},
 'xgboost.XGBClassifier#1': {'n_jobs': -1,
  'n_estimators': 100,
  'max_depth': 3,
  'learning_rate': 0.1,
  'gamma': 0,
  'min_child_weight': 1},
 'mlprimitives.preprocessing.ClassDecoder#1': {}}Or obtaining the corresponding tunable ranges, ready to be used with a tuner:tunable=piex.get_tunable_hyperparameters(template_name)tunable{'mlprimitives.feature_extraction.CategoricalEncoder#1': {'max_labels': {'type': 'int',
   'default': 0,
   'range': [0, 100]}},
 'sklearn.preprocessing.Imputer#1': {'strategy': {'type': 'str',
   'default': 'mean',
   'values': ['mean', 'median', 'most_frequent']}},
 'sklearn.preprocessing.StandardScaler#1': {'with_mean': {'type': 'bool',
   'default': True},
  'with_std': {'type': 'bool', 'default': True}},
 'mlprimitives.preprocessing.ClassEncoder#1': {},
 'xgboost.XGBClassifier#1': {'n_estimators': {'type': 'int',
   'default': 100,
   'range': [10, 1000]},
  'max_depth': {'type': 'int', 'default': 3, 'range': [3, 10]},
  'learning_rate': {'type': 'float', 'default': 0.1, 'range': [0, 1]},
  'gamma': {'type': 'float', 'default': 0, 'range': [0, 1]},
  'min_child_weight': {'type': 'int', 'default': 1, 'range': [1, 10]}},
 'mlprimitives.preprocessing.ClassDecoder#1': {}}Scoring Templates and PipelinesTheS3PipelineExplorerclass also allows cross validating templates and pipelines
over any of the datasets.Scoring a PipelineThe simplest use case is cross validating a pipeline over a dataset.
For this, we must pass the ID of the pipeline and the name of the dataset to the methodscore_pipeline.The dataset can be the one that was used during the experiments or a different one.piex.score_pipeline(pipeline['id'],'185_baseball')(0.6921128080904511, 0.09950216269594728)piex.score_pipeline(pipeline['id'],'uu4_SPECT')(0.8897656842904123, 0.037662864373452655)Optionally, the cross validation configuration can be changedpiex.score_pipeline(pipeline['id'],'uu4_SPECT',n_splits=3,random_state=43)(0.8869488536155202, 0.019475563687443638)Scoring a TemplateA Template can also be tested over any dataset by passing its name, the dataset and, optionally,
the cross validation specification. You have to make sure to choose template that is relevant for
the task/data modality for which you want to use it.If no hyperparameters are passed, the default ones will be used:piex.score_template(template_name,'uu4_SPECT',n_splits=3,random_state=43)(0.8555346666968675, 0.028343173498423108)You can get the default hyperparameters, and update the hyperparameters by setting values
in the dictionary:With this anyone can tune the templates that we have for different task/data modality
types using their own AutoML routine. If you choose to do so, let us know the score you
are getting and the pipeline and we will add to our database.hyperparameters=piex.get_default_hyperparameters(template_name)hyperparameters['xgboost.XGBClassifier#1']['learning_rate']=1piex.score_template(template_name,'uu4_SPECT',hyperparameters,n_splits=3,random_state=43)(0.8754554700753094, 0.019151608028236813)History0.2.0Pin dependency versions to ensure reproducibilityAllow S3PipelineExplorer usage without AWS credentialsFix minor bug in S3PipelineExplorer0.1.1Improved documentationMinor bugfixes and improvements to the explorer API0.1.0First release on PyPI