Package: maggot
maggotis a very simple but useful library with primary goal to remove the need of custom experiment tracking approaches most people typically use. The focus is on reproducibility and removing boilerplate code.Main issuesmaggot(at least partially) solves:Removes the need for meditations on what is a proper name for the experiment. Say you are a machine learning researcher/engineer and you want to train a convolutional neural network with a particular set of parameters, say, 50 convolutional layers, dropout 0.5 and relu activations. You might want to create a separate directory for this experiment to store some checkpoints and summaries there. If you do not expect to have a lot of different models you can simply go off with something like "convnet50layers" or "convnet50relu". But if the number of experiments grows, you need a more reliable and automated solution.maggotoffers such a solution - any experiment you run will have a name derived from the configuration parameters of your model. For the aforementioned model it would be "50-relu-0.5". You still can use a custom experiment name if you want to.Assists reproducibility. Ever experienced a situation when results you got a month ago with an "old" model are no longer reproducible? Even if you are using git, you probably had used some command-line arguments that are now lost somewhere in the bash history...maggotstores all command line parameters, saves full stdout, and much more.Restoring a model is now really painless! Sincemaggotsaves all the parameters you used to run the experiment, all you need to restore a model is to provide a path to a saved experiment.Let's consider a toy example and train an SVM on the Iris dataset.First, import required packages and define command-line arguments:importargparseimportosimportpicklefromsklearn.datasetsimportload_irisfromsklearn.svmimportSVCfromsklearn.model_selectionimportcross_val_score,StratifiedKFoldfrommaggotimportExperimentparser=argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)parser.add_argument("--C",type=float,default=1.0,help="Regularization parameter for SVM")parser.add_argument("--gamma",type=float,default=0.01,help="Kernel parameter for SVM")parser.add_argument("--cv",type=int,default=5,help="Number of folds for cross-validation")parser.add_argument("--cv_random_seed",type=int,default=42,help="Random seed for cross-validation iterator")args=parser.parse_args()Define a configuration object for the experiment:svm_config={"model":{"C":args.C,"gamma":args.gamma},"crossval":{"n_folds":args.cv,"_random_seed":args.cv_random_seed}}Therandom_seedparameter is not really important for analyzing and comparing different experiments, so we included an underscore before its name in config. This tellsmaggotto ignore it for experiment's identifier (short name).Lets create an experiment object!experiment=Experiment(config=svm_config)From here you can reach the model identifier:>>> experiment.config.identifier
5-1.0-0.01Or the experiment directory:>>> experiment.experiment_dir
experiments/5-1.0-0.01Lets examine what this directory contains by now.tree -a experiments/5-1.0-0.01/

experiments/5-1.0-0.01/
└── .maggot
    ├── command
    ├── config.json
    ├── environ
    ├── logs
    │   └── 2020-11-15-14-53-22-1605444802
    └── results.jsonThecommandfile contains the command we run from terminal,config.jsonstores the configuration, andlogsdirectory will store any output you get during the run.Lets train the model!withexperiment:config=experiment.configmodel=SVC(C=config.model.C,gamma=config.model.gamma)score=cross_val_score(model,X=iris.data,y=iris.target,scoring="accuracy",cv=StratifiedKFold(config.crossval.n_folds,shuffle=True,random_state=config.crossval._random_seed),).mean()Note that we can access parameters using dot notation rather than["keyword"]notation, which looks much nicer.We can print accuracy and this will be stored in a log file:print("Accuracy is",round(score,4))Additionaly it's possible to registerscoreas a result of this experiment:experiment.register_result("accuracy",score)This creates aresults.jsonfile in the.maggotdirectory with the following content:{
    "accuracy": 0.9333333333333332
}Later we can use such files from different experiments to be able to compare them.Finally, lets save the model usingpicklemodule.withopen(os.path.join(experiment.experiment_dir,"model.pkl"),"wb")asf:pickle.dump(model,f)See how directory structure has changed:tree -a experiments/5-1.0-0.01/

experiments/5-1.0-0.01/
├── .maggot
│   ├── command
│   ├── config.json
│   ├── environ
│   ├── logs
│   │   └── 2020-11-15-14-53-22-1605444802
│   └── results.json
└── model.pklIf we want to restore the experiment we can easily do:withExperiment(resume_from="experiments/5-1.0-0.01")asexperiment:config=experiment.config# the same config we created above...Configuration file and other stuff is loaded automatically.We can easily run several experiments with different parameters:python ../maggot/examples/iris_sklearn.py --C=10
python ../maggot/examples/iris_sklearn.py --C=10 --gamma=1
python ../maggot/examples/iris_sklearn.py --C=10 --gamma=0.1
python ../maggot/examples/iris_sklearn.py --C=0.001 --gamma=0.1
python ../maggot/examples/iris_sklearn.py --C=0.001 --gamma=10And now let's compare them!maggot summarize experiments --sort accuracy

Results for /home/dmytro/code/stuff/mag-tests/experiments:

              accuracy
5-10.0-0.1    0.986667
5-10.0-0.01   0.973333
5-10.0-1.0    0.953333
5-0.001-0.1   0.926667
5-0.001-10.0  0.813333CLImaggothas a minimalistic CLI interface for working with experiments and being able to inspect them, compare between them and so forth.Currently, the following commands are supported:summarize     Summarize metrics from all experiments in a given directory.
  show-config	Show experiment config.
  show-command	Show command used to run an experiment.
  config-diff	Show diff between configs in two experiments.Simple typemaggot COMMANDin terminal to see help for a specific command.InstallationTo install, clone the repository and then usepip install .or simply runpip install git+https://github.com/ex4sperans/maggot.gitto install directly from GitHub. The repository will be added to PyPI soon to simplify the installation.