Package: memento-ml
MEMENTOMEMENTOis a Python library for running computationally expensive experiments.Running complex sets of machine learning experiments is challenging and time-consuming due to the lack of a unified framework.
This leaves researchers forced to spend time implementing necessary features such as parallelization, caching, and checkpointing themselves instead of focussing on their project.
To simplify the process, we introduceMEMENTO, a Python package that is designed to aid researchers and data scientists in the efficient management and execution of computationally intensive experiments.MEMENTOhas the capacity to streamline any experimental pipeline by providing a straightforward configuration matrix and the ability to concurrently run experiments across multiple threads.If you need to run a large number of time-consuming experimentsMEMENTOcan help:Structure your configurationParallelize experiments across CPUsSave and restore resultsCheckpoint in-progress experimentsSend notifications when experiments fail or finishGetting StartedMEMENTOis officially available on PyPl. To install the package:Installpipinstallmemento-mlThe Configuration MatrixThe core ofMEMENTOis a configurationmatrixthat describes the list of experiments you
wantMEMENTOto run. This must contain a keyparameterswhich is itself a dict, this describes
each paramter you want to vary for your experiments and their values.As an example let's say you wanted to test a few simple linear classifiers on a number of
image recognition datasets. You might write something like this:Don't worry if you're not working on machine learning, this is just an example.matrix={"parameters":{"model":[sklearn.svm.SVC,sklearn.linear_model.Perceptron,sklearn.linear_model.LogisticRegression],"dataset":["imagenet","mnist","cifar10","quickdraw"]}}MEMENTOwould then generate 12 configurations by taking thecartesian productof the
parameters.Frequently you might also want to set some global configuration values, such as a regularization
parameter or potentially even change your preprocessing pipeline. In this caseMEMENTOalso
accepts a "settings" key. These settings apply to all experiments and can be accessed from the
configuration list as well as individual configurations.matrix={"parameters":...,"settings":{"regularization":1e-1,"preprocessing":make_preprocessing_pipeline()}}You can also exclude specific parameter configurations. Returning to our machine learning
example, if you know SVCs perform poorly on cifar10 you might decide to skip that
experiment entirely. This is done with the "exclude" key:matrix={"parameters":...,"exclude":[{"model":sklearn.svm.SVC,"dataset":"cifar10"}]}Running an experimentAlong with a configuration matrix you need some code to run your experiments. This can be anyCallablesuch as a function, lambda, class, or class method.frommementoimportMemento,Config,Contextdefexperiment(context:Context,config:Config):classifier=config.model()dataset=fetch_dataset(config.dataset)classifier.fit(*dataset)returnclassifierMemento(experiment).run(matrix)You can also perform a dry run to check you've gotten the matrix correct.Memento(experiment).run(matrix,dry_run=True)Runningconfigurations:{'model':sklearn.svm.SVC,'dataset':'imagenet'}{'model':sklearn.svm.SVC,'dataset':'mnist'}{'model':sklearn.svm.SVC,'dataset':'cifar10'}{'model':sklearn.svm.SVC,'dataset':'quickdraw'}{'model':sklearn.linear_model.Perceptron,'dataset':'imagenet'}...ExitingduetodryrunCode demoCode demo can be foundhere.MEMENTOdoes not depend on any ML packages, e.g.,scikit-learn. Thescikit-learnandjupyterlabpackages are required to run the demo (./demo/*).pipinstallmemento-mlscikit-learnjupyterlabRoadmapFinish HPC supportImprove result serialisationImprove customization for notificationContributorsZac Pullar-StreckerFeras AlbaroudiLiam Scott-RussellJoshua de WetNipun JastiJames LambertonXinglong (Luke) ChangLiam BrydonIoannis ZiogasKatharina DostJoerg WickerLicenseMEMENTO is licensed under the3-Clause BSD Licenselicense.