Package: mle-toolbox
Lightweight Distributed ML Experiments Management 🛠️Coming up with the right hypothesis is hard - testing it should be easy.ML researchers need to coordinate different types of experiments on separate remote resources. TheMachine Learning Experiment (MLE)-Toolboxis designed to facilitate the workflow by providing a simple interface, standardized logging, many common ML experiment types (multi-seed/configurations, grid-searches and hyperparameter optimization pipelines). You can run experiments on your local machine, high-performance compute clusters (SlurmandSun Grid Engine) as well as on cloud VMs (GCP). The results are archived (locally/GCS bucket) and can easily be retrieved or automatically summarized/reported.What Does Themle-toolboxProvide? 🧑‍🔧API for launching jobs on cluster/cloud computing platforms (Slurm, GridEngine, GCP).Common machine learning research experiment setups:Launching and collecting multiple random seeds in parallel/batches or async.Hyperparameter searches: Random, Grid, SMBO, PBT, Nevergrad, etc.Pre- and post-processing pipelines for data preparation/result visualization.Automated report generation for hyperparameter search experiments.Storage/retrieval of results and database in Google Cloud Storage Bucket.Resource monitoring with dashboard visualization.The 4 Stepmle-toolboxCooking Recipe 🍲Follow theinstructions belowto install themle-toolboxand set up your credentials/configuration.Learn more about the individual infrastructure subpackages with thededicated tutorial.Read thedocsexplaining the pillars of the toolbox & the experiment meta-configuration job.yamlfiles .Check out theexample workflows 📄to get started.Run your own experiment using thetemplate files/projectandmle run.Installation ⏳If you want to use the toolbox on your local machine follow the instructions locally. Otherwise do so on your respective cluster resource (Slurm/SGE). A PyPI installation is available via:pip install mle-toolboxIf you want to get the most recent commit, please install directly from the repository:pip install git+https://github.com/mle-infrastructure/mle-toolbox.git@mainThe Core Toolbox Subcommands 🌱You are now ready to dive deeper into the specifics ofexperiment configurationand can start running your first experiments from the cluster (or locally on your machine) with the following commands:CommandDescription🚀mle runStart up an experiment (multi-config/seeds, search).🖥️mle monitorMonitor resource utilisation (mle-monitorwrapper).📥mle retrieveRetrieve experiment result from GCS/cluster.💌mle reportCreate an experiment report with figures.⏳mle initSetup of credentials & toolbox settings.🔄mle syncExtract all GCS-stored results to your local drive.🗂mle projectInitialize a new project by cloningmle-project.📝mle protocolList a summary of the most recent experiments.You can find more documentation for each subcommandhere.Examples 📄 & Notebook Walkthroughs 📓Job TypesDescription📄Single-Objectivemulti-configs,hyperparameter-searchCore experiment types.📄Multi-Objectivehyperparameter-searchMulti-objective tuning.📄Multi Bashmulti-configsBash-based jobs.📄Quadratic PBThyperparameter-searchPBT on toy quadratic surrogate.📄Hyperbandhyperparameter-searchHyperband on toy polynomial problem.DescriptionColab📓Getting StartedGet started with the toolbox.📓SubpackagesGet started with the toolbox subpackages.📓MLExperimentIntroduction toMLExperimentwrapper.📓EvaluationEvaluation of gridsearch results.📓GIF AnimationsWalk through a set of animation helpers.📓TestingPerform hypothesis tests on logs.Acknowledgements & Citing the MLE-Infrastructure ✏️If you use parts themle-toolboxin your research, please cite it as follows:@software{mle_infrastructure2021github,
  author = {Robert Tjarko Lange},
  title = {{MLE-Infrastructure}: A Set of Lightweight Tools for Distributed Machine Learning Experimentation},
  url = {http://github.com/mle-infrastructure},
  year = {2021},
}Development 👷You can run the test suite viapython -m pytest -vv tests/. If you find a bug or are missing your favourite feature, feel free to create an issue and/or startcontributing🤗.