Package: dswizard
dswizarddswizardis an efficient solver for machine learning (ML) pipeline synthesis inspired by human behaviour. It
automatically derives a pipeline structure, selects algorithms and performs hyperparameter optimization. This repository
contains the source code and data used in our publicationIterative Search Space Construction for Pipeline Structure Search.How to installThe code has only be tested with Python 3.8, but any version supporting type hints should work. We recommend using a
virtual environment.python3 -m virtualenv venv
source venv/bin/activatedswizardis available on PyPI, you can simply install it viapip install dswizardAlternatively, you can checkout the source code and install it directly viapip install -e dswizardNow you are ready to go.Visualizationdswizardcontains an optional pipeline search space visualization functionality intended for debugging and
explainability. If you don't need this feature, you can skip this step. To use the visualization you have to installGraphvizmanually and add the additional visualization libraries usingpip install dswizard[visualization]UsageIn the folder scripts, we have provided scripts to showcase usage ofdswizard. The most important script isscripts/1_optimize.py. This script solves the pipeline synthesis for a given task. To get usage information usepython dswizard/scripts/1_optimize.py --helpyielding a output similar tousage: 1_optimize.py [-h] [--wallclock_limit WALLCLOCK_LIMIT] [--cutoff CUTOFF] [--log_dir LOG_DIR] task

Example 1 - dswizard optimization.

positional arguments:
  task                  OpenML task id

optional arguments:
  -h, --help            show this help message and exit
  --wallclock_limit WALLCLOCK_LIMIT
                        Maximum optimization time for in seconds
  --cutoff CUTOFF       Maximum cutoff time for a single evaluation in seconds
  --log_dir LOG_DIR     Directory used for logging
  --fold FOLD           Fold of OpenML task to optimizeYou have to pass anOpenMLtask id. For example, to create pipelines for thekc2data set
usepython dswizard/scripts/1_optimize.py 3913. Via the optional parameter you can change the total optimization time
(default 300 seconds), maximum evaluation time for a single configuration (default 60 seconds), the directory to store
optimization artifacts (defaultrun/{TASK}) and the fold to evaluate (default 0).The optimization procedure prints the best found pipeline structure with the according configuration and test performance
to the console., similar to2020-11-13 16:45:55,312 INFO     root            MainThread Best found configuration: [('22', KBinsDiscretizer), ('26', PCAComponent), ('28', AdaBoostingClassifier)]
Configuration:
  22:encode, Value: 'ordinal'
  22:n_bins, Value: 32
  22:strategy, Value: 'kmeans'
  26:keep_variance, Value: 0.9145797030897109
  26:whiten, Value: True
  28:algorithm, Value: 'SAMME'
  28:learning_rate, Value: 0.039407336108331845
  28:n_estimators, Value: 138
 with loss -0.8401893431635389
2020-11-13 16:45:55,312 INFO     root            MainThread A total of 20 unique structures where sampled.
2020-11-13 16:45:55,312 INFO     root            MainThread A total of 58 runs where executed.
2020-11-13 16:45:55,316 INFO     root            MainThread Final pipeline:
FlexiblePipeline(configuration={'22:encode': 'ordinal', '22:n_bins': 32,
                                '22:strategy': 'kmeans',
                                '26:keep_variance': 0.9145797030897109,
                                '26:whiten': True, '28:algorithm': 'SAMME',
                                '28:learning_rate': 0.039407336108331845,
                                '28:n_estimators': 138},
                 steps=[('22', KBinsDiscretizer(encode='ordinal', n_bins=32, strategy='kmeans')),
                        ('26', PCAComponent(keep_variance=0.9145797030897109, whiten=True)),
                        ('28', AdaBoostingClassifier(algorithm='SAMME', learning_rate=0.039407336108331845, n_estimators=138))])
2020-11-13 16:45:55,828 INFO     root            MainThread Final test performance -0.8430735930735931Additionally, an ensemble of the evaluated pipeline candidates is constructed.2020-11-13 16:46:06,371 DEBUG    Ensemble        MainThread Building bagged ensemble
2020-11-13 16:46:09,606 DEBUG    Ensemble        MainThread Ensemble constructed
2020-11-13 16:46:10,472 INFO     root            MainThread Final ensemble performance -0.8528138528138528 based on 11 pipelinesIn the log directory (defaultrun/{task}) four files are stored:log.txtcontains the complete logging outputresults.jsoncontains detailed information about all evaluated hyperparameter configurations.search_graph.pdfis a visual representation of the internal pipeline structure graph.structures.jsoncontains all tested pipeline structures including the list of algorithms and the complete configuration space.BenchmarkingTo assess the performance ofdswizardwe have implemented an adapter for the OpenMLautomlbenchmarkavailablehere. Please refer to that repository for benchmarkingdswizard. The
filescripts/2_load_pipelines.py,scripts/3_load_performance.pyandscripts/4_load_trajectories.pyare used to
comparedswizardwithautosklearnandtpot, both also evaluated viaautomlbenchmark.Meta-LearningThe meta-learning base used in this repository is created usingmeta-learning-base.
Please see this repository on how to create the required meta-learning models.For simplicity, we directly provide a random and sgd forest regression model trained on all available data ready to use.
It is available in indswizard/assets/.scripts/1_optimize.pyis already configured to use this model.The data used to train the regression model is also availableonline.
Please refer tometa-learning-baseto see how to train the model
from the raw data.dswizard-componentsThis repository only contains the optimization logic. The actual basic ML components to be optimized are available indswizard-components. Currently, onlysklearncomponents are
supported.Release New VersionIncrease the version number insetup.pyand build a new release withpython setup.py sdist. Finally, upload the
new version usingtwine upload dist/dswizard-<VERSION>.tar.gz.