Package: boilr
boiler-pytorchBasic framework for training stuff in PyTorch. It's quite tailored to projects
I've been working on lately, so it's meant for personal use. Its sole purpose is
to do away withboilrplate code, and having it here makes it easier to
share it across projects.InstallpipinstallboilrUsage example/templateThere's a usage example that can be useful as template. It's a basic VAE
for MNIST quickly hacked together. The example files are:example.pyexample_evaluate.pyexperiments/mnist_experiment/data.pyexperiments/mnist_experiment/experiment_manager.pymodels/mnist_vae.pyInstall requirements and run the example:pipinstall-rrequirements.txtCUDA_VISIBLE_DEVICES=0pythonexample.pyFor evaluation:CUDA_VISIBLE_DEVICES=0pythonexample_evaluate.py--ll--ll-samples100--load$RUN_NAMEusing the name of the folder inoutput/generated from running the example.Quick referenceBuilt-in functionalitiesThe following functionalities are available out-of-the-box:Easy logging of metrics to tensorboard and to a pickle file. Metrics are collected at every training step, smoothed, and logged/saved at a specified frequency. The amount of smoothing is also customizable.Summaries of the metrics are automatically printed after each training and testing phase. This can be easily customized.Training speed, gradient norm (global and per-parameter), and L2 norm of the model parameters are all automatically logged.It's easy to save images from testing, in a dedicated folder.Gradient clipping (by global norm), controllable through a command-line argument.Automatic model checkpointing, with command-line argument to control the maximum number of recent checkpoints to be kept.Command-line argument to resume training from checkpoint, and everything is taken care of.Progress bar for training and testing, usingtqdm. Can be switched off.Data-dependent initialization (command-line argument).Reproducibility: set random seed across all devices and Python libraries.A suite of utility classes and methods in the packagesboilr.nnandboilr.utils(most of them for internal use).
In particularboilr.nn.modulesandboilr.utils.vizmight be more generally useful.A long list of command-line arguments to control some of the behaviour above.
Some arguments are not directly used, but it's convenient to have them already defined: e.g. if a customDataLoaderis necessary, the batch size is easily accessible withargs.batch_size; and when creating the optimizer, the learning rate isargs.lr.Seeboilr.optionsfor package-wide options. Usually it's not necessary to change them, but they give some more flexibility.Command-line argumentsThere are built-in command-line arguments with default values. These defaults can be easily
overridden programmatically when making the experiment class that subclassesboilr's.
The built-in arguments are the following:batch-size: training batch size (default: None)test-batch-size: test batch size (default: None)lr: learning rate (default: None)max-grad-norm: maximum global norm of the gradient. It is clipped if larger. If None, no clipping is performed. (default: None)seed: random seed (default: 54321)tr-log-every: log training metrics every this number of training steps (default: 1000)ts-log-every: log test metrics every this number of training steps. It must be a multiple of--tr-log-every(default: 1000)ts-img-every: save test images every this number of training steps. It must be a multiple of--ts-log-every(default: same as--ts-log-every)checkpoint-every: save model checkpoint every this number of training steps (default: 1000)keep-checkpoint-max: keep at most this number of most recent model checkpoints (default: 3)max-steps: max number of training steps (default: 1e10)max-epochs: max number of training epochs (default: 1e7)nocuda: do not use cuda (default: False)descr: additional description for experiment namedry-run: do not save anything to disk (default: False)resume: load the run with this name and resume trainingAdditionally, forVAEExperimentManager, the following arguments are available:ll-every: evaluate log likelihood (with the importance-weighted bound) every this number of training steps (default: 50000)ll-samples: number of importance-weighted samples to evaluate log likelihood (default: 100)Getting startedsubclass a base dataset manager class;subclass a base model class;subclass a base experiment manager class (the model class is used in here);make a short script that creates the experiment object, uses it to create aboilr.Trainer, and runs the trainer;optionally, subclass the base evaluator to set up an "offline" evaluation pipeline.See below for more details.Dataset manager class (1)The classboilr.data.BaseDatasetManagermust be subclassed. The subclassmustimplement
the method_make_datasetswhich should return a tuple(train, test)with the training
and test sets as PyTorchDatasets.
A basic implementation of_make_dataloadersis already provided, but can be overridden to make
custom data loaders.Model class (2)One of the model classes must be subclassed to inherit core methods in the base implementationboilr.models.BaseModel.
These models also automatically subclasstorch.nn.Module(so it must implementforward).
In addition,boilr.models.BaseGenerativeModel(subclassingBaseModel) defines a methodsample_priorthat must be implemented by subclasses.Experiment manager class (3)One of the base experiment classes inboilr.experimentsmust be subclassed. The subclassmustimplement:_make_datamanagerto create the dataset manager, which should subclassboilr.data.BaseDatasetManager;_make_modelto create the model, which should subclassboilr.models.BaseModel;_make_optimizerto create the optimizer, which should subclasstorch.optim.optimizer.Optimizer;forward_passto perform a simple single-pass model evaluation and returns losses and metrics;test_procedureto evaluate the model on the test set (usually heavily based on theforward_passmethod).Typically should be overridden:_define_args_defaults,_add_args, and_check_args(or a subset of these) to manage parsing of command-line arguments;_make_run_descriptionwhich returns a string description of the run, used for output folders;save_imagesto save output images (e.g. reconstructions and samples in VAEs).May be overridden for additional control:post_backward_callbackis called by theTrainerafter the backward pass but before the optimization step;get_metrics_dicttranslates a dictionary of results to a dictionary of metrics to be logged (by default this simply copies over the keys);train_log_strandtest_log_strreturn log strings for test and training metrics.Note: The classVAEExperimentManagerimplements defaulttest_procedureandsave_imagesmethods for variational inference with VAEs.Example training script (4)fromboilrimportTrainerfrommy_experimentimportMyExperimentClassif__name__=="__main__":experiment=MyExperimentClass()trainer=Trainer(experiment)trainer.run()Offline evaluator class (5)If offline evaluation is necessary,boilr.eval.BaseOfflineEvaluatorcan be subclassed by implementing:runto run the evaluation;as above,_define_args_defaults,_add_args, and_check_args(or a subset of these) to manage parsing of command-line arguments.The methodruncan be executed by simply calling the evaluator object.
Seeexample_evaluate.py.NotesIt also works withouttensorboard, but it won't save tensorboard logs.