Package: dlhpcstarter
Aims of this library:To be simple and easy to understand so that the focus is on the data science.To reduce the time taken from implementation to results.To promote rapid development of models via configuration files, class composition and/or class inheritance.Reduce boilerplate code (sections of code that are repeated in multiple places with little to no variation).To simplify cluster management and distributed computing with High Performance Computing (HPC).Be able to easily accommodate multiple research avenues simultaneously.To cooperatively improve the functionality and documentation of this repository to make it better!Features:TheLightningLightningModuleandTrainerare used to implement, train, and test models. It allows for many of the above aims to be accomplished, such as simplified distributed computing and a reduction of boilerplate code. It also allows us to simply use class inheritance and composition, allowing for rapid development.TheCompose APIofHydrais used to create a hierarchical configuration, allowing for rapid development.Neptune.aiis used to track experiments; metric scores are automatically uploaded toNeptune.ai, allowing you to easily track your experiments from your browser.Scripts for submission to a cluster manager, such asSLURMare written for you. Also, cluster manager jobs are automatically resubmitted and resumed if they haven't finished before the time-limit.InstallationThe Deep Learning and HPC starter pack is available on PyPI:pipinstalldlhpcstarterTable of ContentsPackage mapTasksModelsDevelopment via Model Composition and InheritanceConfiguration YAML files and argparseDevelopment via Configuration FilesNext level: Configuration composition via HydraStages and TrainerTying it all together:main.pyCluster manager and distributed computingMonitoring using Neptune.aiWhere all the outputs go:exp_dirRepository Wish ListPackage mapThe package is structured as follows:├──  dlhpcstarter
│    │
│    ├── tools                     - for all other modules; tools that are repeadetly used.
│    ├── __main__.py               - __main__.py does the following:
│    │                                    1. Reads command line arguments using argparse.
│    │                                    2. Imports the 'stages' function for the task from task/
│    │                                    3. Loads the specified configuration .yaml for the job from 
│    │                                    4. Submits the job (the configuration + 'stages') to the 
│    │                                       cluster manager (or runs it locally if 'submit' is false).
│    └── cluster.py                - contains the cluster management object.
│    └── command_line_arguments.py - argparse for reading command line arguments.
│    └── trainer.py                - contains an optional wrapper for the Lightning Trainer.
│    └── utils.py                  - small utility definitions.TasksTasks can have any name. The name could be based on the data or the type of inference being made. For example:Two tasks have the same data but require different names due to differing predictions, e.g.,MS-COCO DetectionandMS-COCO Caption.Two tasks may have similar predictions but require different names due to differing data, e.g.,MNISTandChinese MNIST.Some publicly available tasks include:Image classification tasks, e.g.,MNIST,CIFAR10,CIFAR100,ImageNet.Object detection tasks, e.g.,MS-COCO Detection.Image captioning detection tasks, e.g.,MS-COCO Caption.Speech recognition tasks, e.g.,LibriSpeech.Chest X-Ray report generation, e.g.,MIMIC-CXR.What does the name do?It is used to separate the outputs of the experiment from other tasks.ModelsPlease familiarise yourself with theLightning LightningModulein order to correctly implement a model:https://pytorch-lightning.readthedocs.io/en/latest/common/lightning_module.htmlA model is created using aLightning LightningModule. Everything we need for the model can be placed in theLightningModule, including commonly used libraries and objects, for example:torch.nn.Module: base class for all neural networks inPyTorch.transformers: a library containing pre-trained Transformer models.torchvision: a library for image pre-processing and pre-trained computer vision models.torch.utils.data.Dataset: an object that processes each instance of a dataset.torch.utils.data.DataLoader: an object that samples mini-batches from atorch.utils.data.Dataset.Note:The data pipeline could be implemented within theLightningModuleor seperately from a model using aLightning LightningDataModule. TheLightningModuleinstance would then have to be given separately to theLightning Trainerinstance.Example:An example model forcifar10is intask/cifar10/model/baseline.py.Development via Model Composition and InheritanceTo promote rapid development of models, one solution is to use class composition and/or inheritance.For example, we may have a baseline that not only includes a basic model, but also the data pipeline:fromlightning.pytorchimportLightningModulefromtorch.utils.dataimportDataLoader,random_splitimporttorchvisionimporttorchclassBaseline(LightningModule):def__init__(self,lr,...,**kwargs):super(Baseline,self).__init__()self.save_hyperparameters()self.lr=lrself.model=torchvision.models.resnet18(...)defsetup(self,stage=None):ifstage=='fit'orstageisNone:train_set=torchvision.datasets.CIFAR10(...)self.train_set,self.val_set=random_split(train_set,[45000,5000])ifstage=='test'orstageisNone:self.test_set=torchvision.datasets.CIFAR10(...)deftrain_dataloader(self,shuffle=True):returnDataLoader(self.train_set,...)defval_dataloader(self):returnDataLoader(self.val_set,...)deftest_dataloader(self):returnDataLoader(self.test_set,...)defconfigure_optimizers(self):optimiser={'optimizer':torch.optim.SGD(self.parameters(),lr=self.lr,momentum=0.9)}returnoptimiserdefforward(self,images):returnself.model(images)deftraining_step(self,batch,batch_idx):images,labels=batchy_hat=self(images)loss=self.loss(y_hat,labels)self.log_dict({'train_loss':loss},...)returnlossdefvalidation_step(self,batch,batch_idx):images,labels=batchy_hat=self(images)loss=self.loss(y_hat,labels)self.val_accuracy(torch.argmax(y_hat['logits'],dim=1),labels)self.log_dict({'val_acc':self.val_accuracy,'val_loss':loss},...)deftest_step(self,batch,batch_idx):images,labels=batchy_hat=self(images)self.test_accuracy(torch.argmax(y_hat['logits'],dim=1),labels)self.log_dict({'test_acc':self.test_accuracy},...)After training and testing the baseline, we may want to improve upon its performance. For example, if we wanted to make the following modifications:Use a DenseNet instead of a ResNet.Use theAdamWoptimiser.Use a warmup learning rate scheduler.All we would need to do is inherit the baseline and make our modifications:fromtransformersimportget_constant_schedule_with_warmupclassInheritance(Baseline):def__init__(self,num_warmup_steps,**kwargs):super(Inheritance,self).__init__(**kwargs)self.save_hyperparameters()self.num_warmup_steps=num_warmup_stepsself.model=torchvision.models.densenet121(...)defconfigure_optimizers(self):optimiser={'optimizer':torch.optim.AdamW(self.parameters(),lr=self.lr)}optimiser['scheduler']={'scheduler':get_constant_schedule_with_warmup(optimiser['optimizer'],self.num_warmup_steps),'interval':'step','frequency':1,}returnoptimiserWe could also construct a model that is the combination of the two via composition. For example, we may want to use everything fromBaseline, but the optimiser fromInheritance:fromlightning.pytorchimportLightningModuleclassComposite(LightningModule):def__init__(self,**kwargs):self.baseline=Baseline(self,**kwargs)defsetup(self,stage=None):self.baseline.setup(stage)deftrain_dataloader(self,shuffle=True):returnself.baseline.train_dataloader(shuffle)defval_dataloader(self):returnself.baseline.val_dataloader()deftest_dataloader(self):returnself.baseline.test_dataloader()defconfigure_optimizers(self):returnInheritance.configure_optimizers(self)# Use configure_optimizers() from Inheritance.defforward(self,images):returnself.baseline.forward(images)deftraining_step(self,batch,batch_idx):returnself.baseline.training_step(batch,batch_idx)defvalidation_step(self,batch,batch_idx):returnself.baseline.validation_step(batch,batch_idx)deftest_step(self,batch,batch_idx):returnself.baseline.test_step(batch,batch_idx)Configuration YAML files and argparseCurrently, there are two methods for giving arguments:Via command line arguments using theargparsemodule.argparsemainly handles paths, development stage flags (e.g., training and testing flags), and cluster manager arguments.Via a configuration file stored inYAMLformat. Can handle all the arguments defined by theargparseplus more, including hyperparameters for the model.The mandatory arguments include:task, the name of the task.config, relative or absolute path to the configuration file (with or without the extension).module, the module that the model definition is housed.definition, the class representing the model.exp_dir, the experiment directory, i.e., where all outputs, including model checkpoints will be saved.monitor, metric to monitor forModelCheckpointandEarlyStopping(optional), as well as test checkpoint loading (e.g., 'val_loss').monitor_mode, whether the monitored metric is to be maximised or minimised ('max' or 'min').taskandconfigmust be given as command line arguments forargparse:dlhpcstarter--configtask/cifar10/config/baseline--taskcifar10module,definition, andexp_dircan be given either as command line arguments, or be placed in the configuration file.For each model of a task, we define a configuration. Hyperparameters, paths, as well as the device configuration can be stored in a configuration file. Configurations are inYAMLformat, e.g.,task/cifar10/config/baseline.yaml.Development via Configuration FilesIf we have the following configuration file for the aforementioned CIFAR10Baselinemodel,task/cifar10/config/baseline.yaml:train:Truetest:Truemodule:task.cifar10.model.baselinedefinition:Baselinemonitor:'val_acc'monitor_mode:'max'lr:1e-3max_epochs:32mbatch_size:32num_workers:5exp_dir:/my/experiment/directorydataset_dir:/my/datasets/directoryAnother way we can improve upon the baseline model, i.e., the baseline configuration, is by modifying its hyperparameters. For example, we can still useBaseline, but alter the learning rate intask/cifar10/config/baseline_rev_a.yaml:train:Truetest:Truemodule:task.cifar10.model.baselinedefinition:Baselinemonitor:'val_acc'monitor_mode:'max'lr:1e-4# modify this.max_epochs:32mbatch_size:32num_workers:5exp_dir:/my/experiment/directorydataset_dir:/my/datasets/directorydlhpcstarter--configtask/cifar10/config/baseline_rev_a--taskcifar10Next level: Configuration composition via HydraIf your new configuration only modifies a few arguments of another configuration file, you can take advantage of the composition feature ofHydra. This makes creatingtask/cifar10/config/baseline_rev_a.yamlfrom the previous section easy. We simply add the arguments fromtask/cifar10/config/baseline.yamlby adding its name to thedefaultslist:defaults:-baseline-_self_lr:1e-4Note that other configuration files are imported with reference to the current configuration path (not the working directory).Please note that groups are not being used, and packages should be placed using@_global_if the configurations being used for composition are not in the same directory.For example, the following would not work with this repository as the arguments inhpc_pathswill be grouped underpaths:defaults:-paths/hpc_paths-_self_train:Truetest:Trueresumable:Truemodule:task.cifar10.model.baselinedefinition:Baselinemonitor:'val_acc'monitor_mode:'max'lr:1e-3max_epochs:3mbatch_size:32num_workers:5To get around this, simply place@_global_to remove the grouping:defaults:-paths/hpc_paths@_global_# changed here to remove "paths" grouping.-_self_train:Truetest:Trueresumable:Truemodule:task.cifar10.model.baselinedefinition:Baselinemonitor:'val_acc'monitor_mode:'max'lr:1e-3max_epochs:3mbatch_size:32num_workers:5This also allows us to organise configurations easily. For example, if we have the following directory structure:├── task
│   └──  cifar10          
│        └── config  
│            ├── cluster
│            │    ├── 2hr.yaml
│            │    └── 24hr.yaml
│            │
│            ├── distributed
│            │    ├── 1gpu.yaml
│            │    ├── 4gpu.yaml
│            │    └── 4gpu4node.yaml
│            │
│            ├── paths
│            │    ├── local.yaml
│            │    └── hpc.yaml
│            │
│            └── baseline.yamlWithtask/cifar10/config/baseline.yamlas:defaults:-cluster/2hr@_global_-distributed/4gpu@_global_-paths/hpc_paths@_global_-_self_train:Truetest:Trueresumable:Truemodule:task.cifar10.model.baselinedefinition:Baselinemonitor:'val_acc'monitor_mode:'max'lr:1e-3max_epochs:3mbatch_size:32num_workers:5Wheretask/cifar10/config/baseline.yamlwill now include arguments from the following example sub-configurations:task/cifar10/config/cluster/2hr.yaml:memory:32GBtime_limit:'02:00:00'venv_path:/path/to/my/venv/bin/activatetask/cifar10/config/distributed/4gpu.yaml:num_gpus:2strategy:ddptask/cifar10/config/paths/hpc.yaml:exp_dir:/path/to/my/experimentsdataset_dir:/path/to/my/datasetSee the following documentation for more information:https://hydra.cc/docs/1.2/tutorials/basic/your_first_app/defaults/https://hydra.cc/docs/1.2/advanced/defaults_list/#composition-orderhttps://hydra.cc/docs/1.2/advanced/overriding_packages/Stages and TrainerIn each task directory is a Python module calledstages.py, which contains thestagesdefinition. This definition takes an object as input that houses the configuration for a job.Typically, the following things happen instages():TheLightningModulemodel is imported via themodelargument, e.g.,fromsrcimportimporterModel=importer(definition=args.definition,module=args.module)model=Model(**vars(args))Seesrc.utils.importerfor a handy function that imports based on strings.ALightning Trainerinstance is created, e.g.,trainer = lightning.pytorch.Trainer(...).The model is trained using trainer:trainer.fit(model).The model is tested using trainer:trainer.test(model).It handles the training and testing of a model for a task by using aLightning Trainer.A helpful wrapper atsrc/trainer.pyexists that passes frequently used and usefulcallbacks,loggers, andpluginsto aLightning Trainerinstance:fromsrc.dlhpcstarter.trainerimporttrainer_instancetrainer=trainer_instance(**vars(args))Place any of the parameters for the trainer detailed athttps://pytorch-lightning.readthedocs.io/en/stable/common/trainer.html#trainer-class-apiin your configuration file, and they will be passed to theLightning Trainerinstance.Tying it all together:dlhpcstarterThis is an overview of what occurs when the entrypointdlhpcstarteris executed, this is not necessary to understand to use the package.dlhpcstarterdoes the following:Gets the command line arguments usingargparse, e.g., arguments like this:dlhpcstarter--configtask.cifar10.config.baseline--taskcifar10Imports thestagesdefinition for the task usingsrc.utils.importer.Reads the configuration.yamland combines it with the command line arguments.Submitsstagesto the cluster manager ifargs.submit = Trueor runsstageslocally. The command line arguments and the configuration arguments are passed tostagesin both cases.Cluster manager and distributed computingThe following arguments are used for distributed computing:ArgumentDescriptionDefaultnum_workersNo. of workers per DataLoader & GPU.1num_gpusNumber of GPUs per node.Nonenum_nodesNumber of nodes (should only be used withsubmit = True).1The following arguments are used to configure a job for a cluster manager (the default cluster manager is SLURM):ArgumentDescriptionDefaultmemoryAmount of memory per node.'16GB'time_limitJob time limit.'02:00:00'submitSubmit job to the cluster manager.NoneresumableResumable training; Automatic resubmission to cluster manager.NoneqosQuality of service.NonebeginWhen to begin the Slurm job, e.g.now+1hour.NoneemailEmail for cluster manager notifications.Nonevenv_pathPath to ''bin/activate'' of a venv.NoneThese can be given as command line arguments:dlhpcstarter--configtask/cifar10/config/baseline--taskcifar10--submit1--num-gpus4--num-workers5--memory32GBOr they can be placed in the configuration.yamlfile:num_gpus:4# Added.num_workers:5# Added.memory:'32GB'# Added.train:Truetest:Truemodule:task.cifar10.model.baselinedefinition:Baselinemonitor:'val_acc'monitor_mode:'max'lr:1e-3max_epochs:32mbatch_size:32num_workers:5exp_dir:/my/experiment/directorydataset_dir:/my/datasets/directoryAnd executed with:dlhpcstarter--configtask/cifar10/config/baseline--taskcifar10--submitTrueIf using a cluster manager, add the path to thebin/activateof your virtual environment:...venv_path:/my/env/name/bin/activate...Monitoring using Neptune.aiSimply sign up athttps://neptune.ai/and add your username and API token to your configuration file:...neptune_username:my_usernameneptune_api_key:df987y94y2q9hoiusadhc9wy9tr82uq408rjw98ch987qwhtr093q4jfi9uwehc987wqhc9qw4uf9w3q4h897324th...ThePyTorch Lightning Trainerwill then automatically upload metrics using theNeptune LoggertoNeptune.ai. Once logged in tohttps://neptune.ai/, you will be able to monitor your task. See here for information about using the online UI:https://docs.neptune.ai/you-should-know/displaying-metadata.Where all the outputs go:exp_dirThe experiments directory is where all your outputs will be saved, including model checkpoints, metric scores. This is also where the cluster manager script, as well as where stderr and stdout are saved.Note: the trial number also sets the seed number for your experiment.***Description to be finished.Repository Wish ListTransfer cluster management over to submitit:https://ai.facebook.com/blog/open-sourcing-submitit-a-lightweight-tool-for-slurm-cluster-computation/Add description about how to usehttps://neptune.ai/.Usehttps://hydra.cc/instead of argparse (or have the option to use either).https://docs.ray.io/en/latest/tune/index.htmlfor hyperparameter optimisation.Notebook examples.