Package: exman
Simple and minimalistic utility to manage many experiments runs and
custom analysis of resultsWhy another custom solution?My job is to do research in Deep Learning and I have dozens of different
experiments. Testing one hypothesis usually required several runs over
parameter grid. Plotting and visualizing results is often ad-hoc and
updating code producing output is a kind of overhead. Instead I decided
to collect all results in Jupyter notebook and create plots kind ofinterest ~ parameters. As I said, plotting that is a separate task
almost every time. Such tools asModelDBprovide you with simple
visualizations so that they can be easily aggregated for model
comparison. Testing a hypothesis is not about model comparison and thus
requires special treatment.Visualizing results became a kind of pain, you had to remember a mappingparameters->results, separating results into different folders
made even more mess. I had really bad experience in visualizations. I
got that all I need was to iterate over folder with results and apply
the same function to it.Installationpipinstall-Ugit+https://github.com/ferrine/exman.git#egg=exman# orpipinstallexmanSimple StartSimple drop in replacement of standardargparse.ArgumentParser#file: main.pyimportexman# you should always use `exman.simpleroot(__file__)` unless you want another dirparser=exman.ExParser(root=exman.simpleroot(__file__))# `root = ./exman` relative to the main fileparser.add_argument(...)You then just add arguments as you did before without any change.Best PracticesError Handling in mainSince 0.0.3 you can use the following context manager. Ifmain()function fails it will be moved toexman/failsimportexman# you should always use `exman.simpleroot(__file__)` unless you want another dirparser=exman.ExParser(root=exman.simpleroot(__file__))# `root = ./exman` relative to the main fileparser.add_argument(...)...if__name__=='__main__':args=parser.parse_args()withargs.safe_experiment:# do your stuffmain(args)Keep Your Repository CleanTo avoid non reproducible results you can ensure you have commited all changes. Exman will take care and will loghashfor the commit anddiffif any. To use these features you should hint the parser with the repo.importexmanparser=exman.ExParser(root=exman.simpleroot(__file__),git=True)# less fragile solution, but works only locallyparser=exman.ExParser(root=exman.simpleroot(__file__),git="/abs/path/to/repo")# an ok solution, if you are sure in the relative pathparser=exman.ExParser(root=exman.simpleroot(__file__),git=os.path.join(os.path.dirname(__file__),"relative","path","goes","here"),git_assert_clean=True# run assertion check before each run. False by default.)In cli of your favorite experiment you can skip the assertion if you want to:pythontrain.py--git-dirty--other-argsOptional ParametersTo avoid issues inreproducing experimentsyou should consider usingexman.optional(type)for optional
argumentsimportexman# you should always use `exman.simpleroot(__file__)` unless you want another dirparser=exman.ExParser(root=exman.simpleroot(__file__))# `root = ./exman` relative to the main fileparser.add_argument('--myarg',type=exman.optional(int))ValidatorsIn simple argparser you cant easily validate multiple arguments, it is
easy in Exman. You can create an informative error messageimportexman# you should always use `exman.simpleroot(__file__)` unless you want another dirparser=exman.ExParser(root=exman.simpleroot(__file__))# `root = ./exman` relative to the main fileparser.add_argument(...)# here `p` stands for initial namespace parsed from argumentsparser.register_validator(lambdap:p.arg1!=p.arg2orp.arg3==p.arg4,# next line will be autoformatted for you using .format'You have provided wrong set of arguments:{arg1},{arg2},{arg3},{arg4}')Advanced validators can raiseexman.ArgumentErrorthat contains a better message than the one in validators functionMarry Pandas with ExmanPandas is a great tool to work with table data. Experiments are the same
data and can be loaded in python. So all you need is to run batch of
experiments and open a Jupyter notebook.importexmanindex=exman.Index(exman.simpleroot('/path/to/main.py'))experiments=index.info()Table has columnstime (datetime64[ns])of experiment androot (pathlib.Path)path to results. Moreover this table has all
other parameters of the experiment. You later can filter/order the
results according to them and have easy-breezy access to results folder
and it’s content.fori,exinexperiments.iterrows():# do some actions# use ex.param for parameters# ex.root / 'plot.png' for file paths...Local ConfigurationYou can store local configuration files in your experiment folder. You
should provide the filename to ExParser as well.importexman# you should always use `exman.simpleroot(__file__)` unless you want another dirparser=exman.ExParser(root=exman.simpleroot(__file__),default_config_files=['local.cfg'])Local configuration stores globally defined default values, they
override defaults set in main fileAuto StructureIf you want argument specific human friendly directory structure you can
tie specific argument names for thatimportexman# you should always use `exman.simpleroot(__file__)` unless you want another dirparser=exman.ExParser(root=exman.simpleroot(__file__),automark=['arg1','constant'])parser.add_argument('--arg1')Later you can see yourmarked folderlooks like thisexman/marked/arg1/<arg1>/constant/<name-of-experiment>/...This can be usefull if you work in a team. Write inmain.pyimportexman# you should always use `exman.simpleroot(__file__)` unless you want another dirparser=exman.ExParser(root=exman.simpleroot(__file__),automark=['user'],# store `user: myuser` content in local.cfgdefault_config_files=['local.cfg'])parser.add_argument('--user')After you’ve done that, your team runs can be stored in a single exman
directory assuming all access rights are correctly set up.exman/marked/user/<username>/constant/<name-of-experiment>/...Directory Structure and CLIIn command line runs will look also the same:python main.py --param1 foo --param2 barThings change if you actually run the program. It dumps all the parsed
parameters combined with defaults into Yaml style file into locationroot/runs/<name-of-experiment>/params.yaml.name-of-experimentis generic and autocreated on the fly. For quick look or search there
are symlinks in theindexfolder e.g.root/index/<name-of-experiment>.yaml. Since a lot of experiments are
created and debugging is sometimes needed, you might want not to create
debug experiments inrunsfolder. For that case you just add--tmpflag and new filed will be written toroot/tmp/<name-of-experiment>folder. That is convenient as you both
do not loose important info about experiment and results and can restore
these symlinks in index by hand if needed.root
|-- runs
|   `-- xxxxxx-YYYY-mm-dd-HH-MM-SS
|       |-- params.yaml
|       `-- ...
|-- fails
|-- index
|   `-- xxxxxx-YYYY-mm-dd-HH-MM-SS.yaml (symlink)
|-- marked
|   `-- <mark>
|       `-- xxxxxx-YYYY-mm-dd-HH-MM-SS (symlink)
|           |-- params.yaml
|           `-- ...
`-- tmp
    `-- xxxxxx-YYYY-mm-dd-HH-MM-SS
        |-- params.yaml
        `-- ...Rerunning experimentIf you want to reproduce an experiment, you can provide source
configuration file in yaml format. For example:pythonmain.py--configroot/index/<name-of-experiment-to-reproduce>.yamlAll the values will be restored from the previous run. You can also
modify old values in--config...usingpythonmain.py--configroot/index/<name-of-experiment-to-reproduce>.yaml--override-param=new_valueIn case you do not want to restore some argument from saved config (it
may be some dynamic setted variable) you should usevolatile=Trueinadd_argument:parser.add_argument('--my_dynamic_id',default=os.environ.get('AUTOSETTED_ID'),volatile=True)Marking experimentsIf you like some experiments you can mark them for easier later access.cd root_of_exman_dir
exman mark <key> <#ex1> [<#ex2> <#ex3> ...]and later in Jupyterindex=exman.Index(exman.simpleroot('/path/to/main.py'))experiments=index.info('<key>')# assuming you work in a team and use best practice adviceuser_experiments=index.info('user/username')Deleting experimentscd root_of_exman_dir
# delete only index
exman delete <#ex1> [<#ex2> <#ex3> ...]
# delete all files
exman delete --all <#ex1> [<#ex2> <#ex3> ...]