Package: expbase
exp-baseexpbaseis a Python framework that implements a commonly used boilerplate for machine learning experiments.
To that end, it takes care of preparing the usual experimental setup for you, and thus significantly reduces the amount
of code you have to write in order to run your experiments.
More precisely,expbaseimplements the following setup tasks:parse command-line args, and create a config object,prepare the results directory,store the used config as JSON file in the results directory,set up logging,launch the training procedure in a separate process, andwait for checkpoints to be created and launch the evaluation of each in a separate process as it is created.InstallationThe packageexpbasecan be installed via pip as follows:pipinstallexpbaseHow-ToStep 1: Define Your Config Classexpbaseusesargmagiqto parse command-line args into a configuration
object, which means that you have to define a config class that describes the user-defined configuration for running an
experiment.
If you should not be familiar withargmagiq, then please read about ithere.
To that end, your configuration class has to extend the classexpbase.BaseConfig,
which defines several configurations that are either used byexpbasedirectly or required for all machine learning
experiments.
(Make sure to familiarize yourself with those.)Step 2: Implement TheTrainingExecutorAnd TheEvaluationExecutorTo provide the code for training and evaluation of your models, you have to implement two classes, one extendingexpbase.TrainingExecutorand one extendingexpbase.EvaluationExecutor.
Those are abstract classes that require you to implement the methods_run_trainingand_run_evaluation,
respectively.
When an experiment is launched, yourTrainingExecutoris run in a separate process, and whenever a training checkpoint
is delivered, your implementation ofEvaluationExecutoris launched (in parallel) in another process in order to
evaluate the same.
In addition to this, bothTrainingExecutorandEvaluationExecutorhave a method_init, which may be overridden to
run code that would usually go into__init__.
The reason why we recommend using_initinstead of__init__is that instances might be moved between processes
before_run_*is invoked, which occasionally results in issues, if certain objects are created in the
constructor.
However,_initis guaranteed to be invoked just before and in the same process as_run_*.All created instances ofTrainingExecutorandEvaluationExecutorare provided with an instance of your config class
that has been populated based on the given command-lines args.
Both classes store this config object in the attribute_conf, which you can use to access the user-defined
configuration of an experiment from your implementations of_initand_run_*.For additional details, please have a look at the docstrings ofexpbase.TrainingExecutorandexpbase.EvaluationExecutor.Step 3: Launch The ExperimentOnce you defined all required classes, launching an experiment is as easy as creating an instance ofexpbase.Experiment, and
invoking it'srunmethod:importexpbaseasxbxb.Experiment(YourTrainingExecutorClass,YourEvaluationExecutorClass,YourConfigClass,app_name,app_description).run()In this code snippet,app_nameandapp_descriptionare two strings that define the name of your application, which
is used in its synopsis (the usage instruction at the beginning of the help), as well as a description of the same,
which is displayed as part of the help text.Delivering CheckpointsAs pointed out above already, evaluation is launched automatically (in a parallel process) whenever a checkpoint is
delivered.
In order to deliver a checkpoint, the classexpbase.TrainingExecutor(which you are extending with your own training code) provides the method_deliver_ckpt, which expects a singlestrargument that allows for locating an according checkpoint file, usually a file path.
As it depends on the particular experiment, you have to create the checkpoint file in your implementation of_run_trainingbefore you invoke_deliver_ckpt.
The checkpoint string is then provided to theEvaluationExecutorthat is launched to evaluate the according
checkpoint and stored in the attribute_ckpt.
Therefore, your own implementation ofexpbase.EvaluationExecutor.
(more precisely, your_initand_run_evaluation) will find the checkpoint string stored inself._ckpt.Running TestsIf you just want to run a test (without training), for example, to evaluate your model on the test partition, then you
can use the command-line option--test, which causesexpbaseto launch only a singleEvaluationExecutor.
Notice that, in this case, the attribute_ckptof yourEvaluationExecutorwill beNone.
Instead, however, you are required to manually specify a checkpoint when you run the application using the option--ckpt <CKPT>, which is then accessible via the config, i.e.,self._conf.ckpt.Results Directoryexpbasecreates a results directory for you, which is also used to store any files that are generated automatically
(e.g., log files, config, etc.).
By default,./results/is used as results directory, and can be adjusted by means of the arg--results-dir.
Notice further that, for every training run, a subdirectory is created within the results dir, named according to when
the experiment was conducted.
For example, when you run an experiment with--results-dir ./fancy-results, then results will be stored in./fancy-results/2020-06-22--10-11-28(using the actual timestamp, of course).
Please notice that the config object passed toTrainingExecutorandEvaluationExecutors contains the actual
results directoryincludingthe particular subdirectory.Finally, you can (and should) use a title to be appended to your result directories in order to make them easier to
find.
To that end, use the arg--titleto define a label to be appended to the directory that your experiment writes results
to.
For example,--results-dir ./fancy-results --title project-ice-creamdefines the results directory to be./fancy-results/2020-06-22--10-11-28--project-ice-cream(again, using the actual timestamp, of course).Config FileTo make it easier to rerun a conducted experiment,expbasestores the configuration that was used to run one as a
JSON file namedconfig.jsonin the results directory.
This file is produced by means ofargmagiq, and can thus be used
directly to launch the same experiment once again.
For example, if you ran an experiment from the terminal using the command$pythonmain.py--results-dirresults/--seed12345...then you can rerun it with the exact same configuration using$pythonmain.py--path/to/the/config.jsonNotice that, for experiments with a lot of configuration, it might be easier to only use config files in the first
place.Loggingexpbaseuses the packagestreamtologgerto automatically log all
outputs produced viaprint("...")or by an exception, and stores them in three different log files in the results
directory:experiment.logis a top-level log file used by the controller of the experiment,train.logcaptures outputs produced by theTrainingExecutor, andeval.logstores outputs produced byEvaluationExecutors.If you, for some reason, wish you to change the names of those files, you can use the options--general-log,--training-log, and--evaluation-logto do so.