Package: ride
Training wheels, side rails, and helicopter parent for your Deep Learning projects inPyTorch.pipinstallrideZERO-boilerplate AI researchRideprovides a feature-rich, battle-tested boilerplate, so that you can focus on the model-building and research. 🧪Out of the box,Ridegives you:Training and testing methods🏋️‍♂️Checkpointing✅Metrics📈Finetuning schemes👌Feature extraction📸Visualisations👁Hyperparameter search📊Logging📜Command-line interface💻Multi-gpu, multi-node handling via... and moreBoilerplate inheritanceWithRide, we inject functionality by means ofinheritance.
The same way, your network would usually inherit fromtorch.nn.Module, we canmix ina plethora of functionality by inheriting from theRideModule(which also includes thetorch.nn.Module).
In addition, boiler-plate for wiring up optimisers, metrics and datasets can be alsomixed inas seen below.Complete project definition# simple_classifier.pyimporttorchimportrideimportnumpyasnpfrom.examplesimportMnistDatasetclassSimpleClassifier(ride.RideModule,ride.SgdOneCycleOptimizer,ride.TopKAccuracyMetric(1,3),MnistDataset,):def__init__(self,hparams):# `self.input_shape` and `self.output_shape` were injected via `MnistDataset`self.l1=torch.nn.Linear(np.prod(self.input_shape),self.hparams.hidden_dim)self.l2=torch.nn.Linear(self.hparams.hidden_dim,self.output_shape)defforward(self,x):x=x.view(x.size(0),-1)x=torch.relu(self.l1(x))x=torch.relu(self.l2(x))returnx@staticmethoddefconfigs():c=ride.Configs()c.add(name="hidden_dim",type=int,default=128,strategy="choice",choices=[128,256,512,1024],description="Number of hidden units.",)returncif__name__=="__main__":ride.Main(SimpleClassifier).argparse()The above is thecompletecode for a simple classifier on the MNIST dataset.All of the usual boiler-plate code has beenmixed inusing multiple inheritance:RideModuleis a base-module which includespl.LightningModuleand makes some behind-the-scenes python-magic work. For instance, it modifies your__init__function to automatically initiate all the mixins correctly. Moreover, it mixes intraining_step,validation_step, andtest_step.SgdOneCycleOptimizermixes in aconfigure_optimizersfunctionality with SGD andOneCycleLR scheduler.TopKAccuracyMetricadds top1acc and top3acc metrics, which can be used for checkpointing and benchmarking.MnistDatasetmixes intrain_dataloader,val_dataloader, andtest_dataloaderfunctions for theMNIST dataset. Dataset mixins always provideinput_shapeandoutput_shapeattributes, which are handy for defining the networking structure as seen in__init__.ConfigsIn addition to inheriting lifecycle functions etc., the mixins also addconfigsto your module (powered byco-rider).
These define all of the configurable (hyper)parameters including theirtypedefaultvaluedescriptionin plain text (reflected in command-line interface),choicesdefines accepted input rangestrategyspecifies how hyperparameter-search tackles the parameter.Configs specific to the SimpleClassifier can be added by overloading theconfigsmethods as shown in the example.The final piece of sorcery is theMainclass, which adds a complete command-line interface.Command-line interface 💻Train and test$pythonsimple_classifier.py--train--test--learning_rate0.01--hidden_dim256--max_epochs1Example output:lightning:Globalseedsetto123ride:RunningonhostHostName
ride:⭐️Viewprojectrepositoryathttps://github.com/UserName/project_name/tree/commit_hash
ride:Rundataissavedlocallyat/Users/UserName/project_name/logs/run_logs/your_id/version_1
ride:LoggingusingTensorboard
ride:💾Saving/Users/au478108/Projects/ride/logs/run_logs/your_id/version_1/hparams.yaml
ride:🚀Runningtraining
ride:✅Checkpointingonval/losswithoptimisationdirectionmin
lightning:GPUavailable:False,used:False
lightning:TPUavailable:False,using:0TPUcores
lightning:|Name|Type|Params
--------------------------------0|l1|Linear|200K1|l2|Linear|2.6K--------------------------------203KTrainableparams0Non-trainableparams203KTotalparams0.814Totalestimatedmodelparamssize(MB)lightning:Globalseedsetto123Epoch0:100%|████████|3751/3751[00:20<00:00,184.89it/s,loss=0.785,v_num=9,step_train/loss=0.762]lightning:Epoch0,globalstep3437:val/lossreached0.77671(best0.77671),savingmodelto"/Users/UserName/project_name/logs/run_logs/your_id/version_1/checkpoints/epoch=0-step=3437.ckpt"astop1lightning:Savinglatestcheckpoint...
Epoch0:100%|████████|3751/3751[00:20<00:00,184.65it/s,loss=0.785,v_num=9,step_train/loss=0.762]ride:🚀RunningevaluationontestsetTesting:100%|████████|625/625[00:01<00:00,358.86it/s]-------------------------------------
DATALOADER:0TESTRESULTS{'loss':0.7508705258369446,'test/loss':0.7508705258369446,'test/top1acc':0.7986000180244446,'test/top3acc':0.8528000116348267}-------------------------------------

ride:💾Saving/Users/UserName/project_name/logs/run_logs/your_id/version_1/test_results.yamlFeature extraction and visualisationExtract features after layerl1and visualise them withUMAP.$pythonsimple_classifier.py--train--test--extract_features_after_layer="l1"--visualise_features="umap"Example output:Confusion matrix visualisationPlot the confution matrix for the test set.$pythonsimple_classifier.py--train--test--test_confusion_matrix1Example output:Advanced model finetuningLoad model and finetune withgradual unfreeze and discriminative learning rates$pythonsimple_classifier.py--train--finetune_from_weightsyour/path.ckpt--unfreeze_layers_initial1--unfreeze_epoch_step1--unfreeze_from_epoch0--discriminative_lr_fraction0.1Hyperparameter optimizationIf we want to performhyperparameter optimisationacross four gpus, we can run:$pythonsimple_classifier.py--hparamsearch--gpus4Curretly, we useRay Tuneand theASHAalgorithm under the hood.Profile modelYou can check thetimingandFLOPsof the model with:$pythonsimple_classifier.py--profile_modelExample output:Results:flops:203530machine:cpu:architecture:x86_64cores:physical:6total:12frequency:2.60GHzmodel:Intel(R)Core(TM)i7-9750HCPU@2.60GHzgpus:nullmemory:available:5.17GBtotal:16.00GBused:8.04GBsystem:node:d40049release:19.6.0system:Darwinparams:203530timing:batch_size:16num_runs:10000on_gpu:falsesamples_per_second:88194.303+/-17581.377[20177.049,113551.377]time_per_sample:12.031us+/-3.736us[8.807us,49.561us]Additional optionsFor additional configuration options, check out the help:$pythonsimple_classifier.py--helpTruncated output:Flow:Commandsthatcontrolthetop-levelflowoftheprogramme.--hparamsearchRunhyperparametersearch.Thebesthyperparameterswillbeusedforsubsequentlifecyclemethods--trainRunmodeltraining--validateRunmodelevaluationonvalidationset--testRunmodelevaluationontestset--profile_modelProfilethemodel

General:Settingsthatapplytotheprogrammeingeneral.--idIDIdentifierfortherun.Ifnotspecified,thecurrenttimestampwillbeused(Default:202101011337)--seedSEEDGlobalrandomseed(Default:123)--logging_backend{tensorboard,wandb}Typeofexperimentlogger(Default:tensorboard)...

PytorchLightning:Settingsinheritedfromthepytorch_lightning.Trainer...--gpusGPUSnumberofgpustotrainon(int)orwhichGPUstotrainon(listorstr)appliedpernode...

Hparamsearch:Settingsassociatedwithhyperparameteroptimisation...

Module:SettingsassociatedwiththeModule--loss{mse_loss,l1_loss,nll_loss,cross_entropy,binary_cross_entropy,...}Lossfunctionusedduringoptimisation.(Default:cross_entropy)--batch_sizeBATCH_SIZEDataloaderbatchsize.(Default:64)--num_workersNUM_WORKERSNumberofCPUworkerstousefordataloading.(Default:10)--learning_rateLEARNING_RATELearningrate.(Default:0.1)--weight_decayWEIGHT_DECAYWeightdecay.(Default:1e-05)--momentumMOMENTUMMomentum.(Default:0.9)--hidden_dimHIDDEN_DIM{128,256,512,1024}Numberofhiddenunits.(Defualt:128)--extract_features_after_layerEXTRACT_FEATURES_AFTER_LAYERLayernameafterwhichtoextractfeatures.Nestedlayersmaybeselectedusingdot-notation,e.g.`block.subblock.layer1`(Default:)--visualise_features{,umap,tsne,pca}Visualiseextractedfeaturesusingselecteddimensionalityreductionmethod.Visualisationsarecreatedonlyduringevaluation.(Default:)--finetune_from_weightsFINETUNE_FROM_WEIGHTSPathtoweightstofinetunefrom.Allowedextensioninclude{'.ckpt','.pyth','.pth','.pkl','.pickle'}.(Default:)--unfreeze_from_epochUNFREEZE_FROM_EPOCHNumberofepochstowaitbeforestartinggradualunfreeze.If-1,unfreezeisomitted.(Default:-1)--test_confusion_matrix{0,1}Createandsaveconfusionmatrixfortestdata.(Default:0)...Though the above--helpprintout was truncated for readibility, there's still a lot going on!
The general structure is a follows:
First, there are flags for controlling the programme flow (e.g. whether to run hparamsearch or training), then some general parameters (id, seed, etc.), all the parameters from Pytorch Lightning, hparamsearch-related arguments, and finally the Module-specific arguments, which we either in theSimpleClassifieror inherited from the RideModule and mixins.EnvironmentPer default,Rideprojects are oriented around the current working directory and will save logs in the~/logs folders, and cache to~/.cache.This behaviour can be overloaded by changing of the following environment variables (defaults noted):ROOT_PATH="~/"CACHE_PATH=".cache"DATASETS_PATH="datasets"# Dir relative to ROOT_PATHLOGS_PATH="logs"# Dir relative to ROOT_PATHRUN_LOGS_PATH="run_logs"# Dir relative to LOGS_PATHTUNE_LOGS_PATH="tune_logs"# Dir relative to LOGS_PATHLOG_LEVEL="INFO"# One of "DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"ExamplesLibrary Examples:SimpleClassifierMNIST DataloaderCommunity Examples:I3DR(2+1)DSlowFastCoSlowX3DCoX3DCitationBibTeXIf you useRidefor your research and feel like citing it, here's a BibTex:@article{hedegaard2021ride,title={Ride},author={Lukas Hedegaard},journal={GitHub. Note: https://github.com/LukasHedegaard/ride},year={2021}}Badge.MD[![Ride](https://img.shields.io/badge/Built_to-Ride-643DD9.png)](https://github.com/LukasHedegaard/ride).HTML<a href="https://github.com/LukasHedegaard/ride">
  <img src="https://img.shields.io/badge/Built_to-Ride-643DD9.png" height="20">
</a>