Package: experi
ExperiA framework for running command line applications with a range of different
variables.Documentation is available onRead the DocsHow to useActually running experi is a simple process, in a directory with anexperiment.ymlfile run the command$ experiIf for whatever reason you want to name the file something other thanexperiment.ymlor to run a file in a different directory a custom file can be
specified with the-fflag$ experi -f not_an_experiment.ymlNote that since this is designed to keep the specification of the experiment
with the results, the commands will be run in the same directory as the
specified file.The complicated part of getting everything running is the specification of the
experiment in theexperiment.ymlfile. The details on configuring this file is available in thedocumentation.Why should I use this?When running a series of experiments it can be difficult to remember the exact
parameters of the experiment, or even how to run the simulation again.
Additionally for a complex experiment with many variables, iterating through
all the combinations of variables can be unwieldy, error prone, and plain
frustrating.Experi aims to keep all the information about running an experiment in anexperiment.ymlfile which sits in the same directory as the experiment.
Supporting complex iteration of variables incorporated into easily the easily
readable yaml syntax, it is easy to quickly understand the experimental
conditions. Additionally by keeping the configuration file with the results
there is a quick reference to the experimental conditions and replication is as
simple as runningexperi.For more information I have written ablog postwhich goes
into more depth on how this tool has helped my workflow.Project GoalsThe primary goals of this project detailed below. They act as the guiding
principles for the design decisions which are made.Human centricInteractions should be simple, intuitive, and frictionlessShouldn't need to constantly consult documentation to useMinimal expertise required to understandSensible DefaultsTesting a job on a scheduler should be simple, requiring a minimal specificationFast ErrorsErrors in the code should be picked up as soon as possible, i.e. shouldn't arise
after waiting in the job queue.Allow for testing locally using the shell, before running on HPCWhere current functionality doesn't meet these goals please raise an issue, I am
more than happy to discuss. Although do note that these goals are somewhat
opinionated.What about ...?Sumatrais a tool for managing and tracking projects,
with a focus on running a single experiment at a time and the
reproducibility of the results. Experi is more about running many
simulations with a range of parameters, the reproducibility aspect is
a byproduct of the way these parameters are specified. Also Sumatra does
a much better job of the reproducibility than experi, capturing version
numbers and executable paths.SciPipeis a workflow manager similar toSciLuigi,Airflowor any
number of other examples. These tools can be incredibly powerful,
specifying complex networks of dependent tasks and managing their
completion. However, they have a learning curve and can be difficult to
configure with a task scheduler on a HPC. Experi is about simplicity;
taking the workflow you already use and making it more powerful. Experi
also uses the task scheduler for the management of dependent jobs, albeit
the functionality is currently very basic.Snakemakeis a workflow management tool, very similar toGNU Makewhich
supports submitting jobs to a HPC scheduler. I personally have no experience
using it, however from reading the documentation it is a highly configurable
tool with far more functionality than Experi. Experi is more suitable is the
handling of complex specification of variables and using the scheduler for
control of scheduling.InstallationExperi is currently compatible with python>==3.6pip3installexperiNote that for the commandexperito work the directory containing the
executable needs to be in thePATHvariable. In most cases this will probably
be$HOME/.local/binalthough this is installation dependent. If you don't
know where the executable is, on *nix systems the commandfind$HOME-nameexperiwill search everywhere in your home directory to find it. Alternatively
replacing$HOMEwith/will search everywhere.For installation from sourcegitclonehttps://github.com/malramsay64/experi.gitcdexperi
pip3install.To install a development version,pipenvis required which can be installed
by runningpip3installpipenvand installing the dependencies by runningpipenvinstall--dev--threewhich will create a virtual environment for the project. Activating the
virtualenv is can be done by runningpipenvshellwhich creates a new shell with the environment activated. Alternatively
a single command (like the test cases) can be run usingpipenvrunpytestFor those of you trying to run this on a cluster with only user privileges
including the--userflag will resolve issues with pip requiring elevated
permissions installing to your home directory rather than for everyone.pip3 install --user experiFor more information documentation is available onRead the Docs.