Package: assembled
AssembledAssembled is planed to be a framework for ensemble evaluation. It shall run, benchmark, and evaluate ensemble techniques
without the overhead of training base models.Currently, its main features are:Metatasks: A metatasks is a meta-dataset and a class interface. Metatasks contain the predictions and
confidences (e.g. sklearn's predict_proba) of specific base models and the data of an original (OpenML) task.
Moreover, its class interface contains several useful method to simplify the evaluation and benchmarking of ensemble
techniques. A collection of metatasks can be used to benchmark ensemble techniques without the computational overhead
of training and evaluating base models.Assembled-OpenML: an extension of Assembled to build metatasks with data from OpenML. Only a OpenML Task ID must
be passed to the code to generate a metatask for a specific OpenML task. Technically, any ID can be passed. In
practice, only supervised classification tasks are supported so far. Moreover, this tool was build for and tested
against curated benchmarks (like tasks in OpenMLCC-18). Other classification tasks should be supported as well but
bugs might be more likely.FakedClassifiers: Code to simulate the behavior of a base model by passing appropriate data to it during the
initialization. Allows us to evaluate most ensemble techniques without code changes to the original implementation's
code.Supporting Ensemble TechniquesWe created code to make ensemble techniques usable with (pre-fitted) base models.
This is not part of Assembled itself but rather additional example on how to use ensemble techniques with Assembled.
Some implementation support base models by default other do not. See/ensemble_techniques/for more details.Currently, is main use-cases are:Ensembles After AutoML (Post-Processing)This repository/branch also contains the Assembled-OpenML extension.Assembled-OpenMLFor the original code of the workshop paper on Assembled-OpenML, see theautoml_workshop_paperbranchAssembled-OpenML builds Metatasks from OpenML. In this first version of Assembled-OpenML, the predictions correspond to
the top-n best runs (configurations) of an OpenML task. It shall simulate the use case of post-processing an AutoML
tool's top-n set of configurations.Assembled-OpenML enables the user to quickly generate a benchmark set by entering a list of OpenML Task IDs as input
(see our code examples). In general, Assembled-OpenML is an affordable/efficient alternative to creating benchmarks by
hand. It is affordable/efficient, because you do not need to train and evaluate the base models but can directly
evaluate ensemble techniques.InstallationTo install Assembled and Assembled-OpenML, use:pipinstallassembled[openml]If you only want to use Assembled, leave away[openml].To install the newest version (from the main branch), use:pipinstallgit+https://github.com/ISG-Siegen/assembled.git#egg=assembled[openml]Other InstallationsFor experiments, work-in-progress code, or other non-packaged code stored in this repository, we
providerequirements.txtfiles. These can be used to re-create the environments needed for the code.An example workflow for the installation on Linux is:gitclonehttps://github.com/ISG-Siegen/assembled.gitcdassembled
python3-mvenvvenv_assembledsourcevenv_assembled/bin/activate
pipinstall-rrequirements.txtPlease be aware that any relevant-enough subdirectory keeps track of its own requirements through arequirements.txt.
Hence, if you want to use only parts of this project, it might be a better idea to only install the requirements of the
code that you want to use.UsageTo see the example usage of Assembled-OpenML, see the./examples/directory for code examples and more details.A simple example of using Assembled-OpenML to get a Metatask and using Assembled to evaluate an ensemble technique on
the Metatask is:fromassembledopenml.openml_assemblerimportOpenMLAssemblerfromassembled.ensemble_evaluationimportevaluate_ensemble_on_metatask# Import an adapted version of auto-sklearn's Ensemble Selection# (requires the ensemble_techniques directory to be in your local directory)fromensemble_techniques.autosklearn.ensemble_selectionimportEnsembleSelectionfromensemble_techniques.util.metricsimportOpenMLAUROC# -- Use Assembled-OpenML to build a metatask for the OpenML task with ID 3omla=OpenMLAssembler(nr_base_models=50,openml_metric_name="area_under_roc_curve")mt=omla.run(openml_task_id=3)# -- Benchmark the ensemble technique on the metatasktechnique_run_args={"ensemble_size":50,"metric":OpenMLAUROC}fold_scores=evaluate_ensemble_on_metatask(mt,EnsembleSelection,technique_run_args,"autosklearn.EnsembleSelection",pre_fit_base_models=True,meta_train_test_split_fraction=0.5,meta_train_test_split_random_state=0,return_scores=OpenMLAUROC)print(fold_scores)print("Average Performance:",sum(fold_scores)/len(fold_scores))LimitationsRegression is not supportedso far as OpenML has not enough data (runs) on Regression tasks. Would require some
additional implementations.Assembled-OpenML ignores OpenML repetitions (most runs/datasets do not provide repetitions).The file format for the predictions file is not fully standardized in OpenML and hence requires manually adjustment to
all used formats. Hopefully, we found most of the relevant formats with Assembled-OpenML.Some files, which store predictions, seem to have malicious or corrupted predictions/confidence values. If we can not
fix such a case, we store these bad predictors in the Metatask object to be manually validated later on. Moreover,
these predictors can be filtered from the Metatask if one wants to (we do this for every example or experiment).A Comment on Validation DataBy default, and by design, Metatask created only from OpenML data do not have inner fold validation data. To train an
ensemble techniques on metataks created only from OpenML data, we split a fold's predictions on the fold's test data of
into ensemble_train and ensemble_test data. With ensemble_train being used to build/train the ensemble and ensemble_test
being used to evaluate the ensemble.Alternatively, if a metatask and the base models stored in the metatask were initialized / created with validation data,
we can also use the validation data to train the ensemble technique and then test it on all test data/predictions of a
fold.Relevant publicationsIf you use Assembled or Assembled-OpenML in scientific publications, we would appreciate citations.Assembled-OpenML: Creating Efficient Benchmarks for Ensembles in AutoML with OpenML,Lennart Purucker and Joeran
Beel,First Conference on Automated Machine Learning (Late-Breaking Workshop), 2022Linkto publication.@inproceedings{purucker2022assembledopenml,
    title={Assembled-Open{ML}: Creating Efficient Benchmarks for Ensembles in Auto{ML} with Open{ML}},
    author={Lennart Purucker and Joeran Beel},
    booktitle={First Conference on Automated Machine Learning (Late-Breaking Workshop)},
    year={2022}
}