Package: ptpt
Alex's PyTorch Personal Trainer (ptpt)(name subject to change)This repository contains my personal lightweight framework for deep learning
projects in PyTorch.Disclaimer: this project is very much work-in-progress. Although technically
useable, it is missing many features. Nonetheless, you may find some of the
design patterns and code snippets to be useful in the meantime.InstallationInstall frompipby runningpip install ptptYou can also build from source. Simply runpython -m buildin the root of the
repo, then runpip installon the resulting.whlfile.UsageImport the library as with any other python library:fromptpt.trainerimportTrainer,TrainerConfigfromptpt.logimportdebug,info,warning,error,criticalThe core of the library is thetrainer.Trainerclass. In the simplest case,
it takes the following as input:net:a`nn.Module`thatisthemodelwewishtotrain.loss_fn:afunctionthattakesa`nn.Module`andabatchasinput.itreturnsthelossandoptionallyothermetrics.train_dataset:thetrainingdataset.test_dataset:thetestdataset.cfg:a`TrainerConfig`instancethatholdsallhyperparameters.Once this is instantiated, starting the training loop is as simple as callingtrainer.train()wheretraineris an instance ofTrainer.cfgstores most of the configuration options forTrainer. See the class
definition ofTrainerConfigfor details on all options.ExamplesAn example workflow would go like this:Define your training and test datasets:transform=transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.1307,),(0.3081,))])train_dataset=datasets.MNIST('../data',train=True,download=True,transform=transform)test_dataset=datasets.MNIST('../data',train=False,download=True,transform=transform)Define your model:# `Net` could be any `nn.Module`net=Net()Define your loss function that callsnet, taking the full batch as input:# minimising classification errordefloss_fn(net,batch):X,y=batchlogits=net(X)loss=F.nll_loss(logits,y)pred=logits.argmax(dim=-1,keepdim=True)accuracy=100.*pred.eq(y.view_as(pred)).sum().item()/y.shape[0]returnloss,accuracyOptionally create a configuration object:# see class definition for full list of parameterscfg=TrainerConfig(exp_name='mnist-conv',batch_size=64,learning_rate=4e-4,nb_workers=4,save_outputs=False,metric_names=['accuracy'])Initialise the Trainer class:trainer=Trainer(net=net,loss_fn=loss_fn,train_dataset=train_dataset,test_dataset=test_dataset,cfg=cfg)Optionally, register some callback functions:defcallback_fn(_):info("Congratulations, you have completed an epoch!")trainer.register_callback(CallbackType.TrainEpoch,callback_fn)Calltrainer.train()to begin the training looptrainer.train()# Go!See more exampleshere.Weights and Biases IntegrationWeights and Biases logging is supported via theptpt.wandb.WandConfigdataclass.Currently only supports a small set of features:class WandbConfig:
    project: str = None         # project name
    entity: str = None          # wandb entity name
    name: str = None            # run name (leave blank for random two words)
    config: dict = None         # hyperparameters to save on wandb
    log_net: bool = False       # whether to use wandb to watch network gradients
    log_metrics: bool = True    # whether to use wandb to report epoch metricsIf you want to log something else in addition to epoch metrics, you can useptpt.callbacksand access wandb throughtrainer.wandb. When calling log
here, ensure commit is set toFalseto avoid advancing the global step.MotivationI found myself repeating a lot of same structure in many of my deep learning
projects. This project is the culmination of my efforts refining the typical
structure of my projects into (what I hope to be) a wholly reusable and
general-purpose library.Additionally, there are many nice theoretical and engineering tricks that
are available to deep learning researchers. Unfortunately, a lot of them are
forgotten because they fall outside the typical workflow, despite them being
very beneficial to include. Another goal of this project is to transparently
include these tricks so they can be added and removed with minimal code change.
Where it is sane to do so, some of these could be on by default.Finally, I am guilty of forgetting to implement decent logging: both of
standard output and of metrics. Logging of standard output is not hard, and
is implemented using other libraries such asrich.
However, metric logging is less obvious. I'd like to avoid larger dependencies
such as tensorboard being an integral part of the project, so metrics will be
logged to simple numpy arrays. The library will then provide functions to
produce plots from these, or they can be used in another library.TODO:Add arbitrary callback support at various points of executionAdd metric trackingAdd more learning rate schedulersAdd more optimizer optionsAdd logging-to-fileAdds silent and simpler loggingSupport for distributed / multi-GPU operationsSet of functions for producing visualisations from disk dumpsGeneral suite of useful functionsReferencesrichby@willmcguganCitations