Package: tortto
PyTorttoIntroInstallationPrerequisitesQuick startExamplesResnetUNetVision TransformerDCGANYOLOv1IntroThis is a pytorch style machine learning framework implemented entirely in numpy, with GPU acceleration from cupy.Similar to the pokemon "ditto", pytortto works exactly like pytorch, although inferior in terms of speed. The purpose of this project is to understand how deep learning algorithms and frameworks like pytorch work under the hood. Max effort was given to correctness, calculation efficiency (like simpler Jacobian in logsoftmax, efficient implementation of convolution etc.), numerical stability (log-sum-exp used in logsigmoid, BCEWithLogitsLoss etc.), and memory efficiency (implementation of caching, view etc.).When computed in GPU, Tortto is around 1.5(vision transformers) ~ 3(CNNs) times slower than pytorch. It also achieves the same complexity as pytorch, which means tortto can be used to train relatively larger models such asresnet101and vision transformerViT-B/16with the same speed ratio.Tortto implements reverse mode automatic differentiation and supports dynamic computation graph like pytorch.InstallationpipinstalltorttoNew in version 1.3Computation graph now uses operations as nodes, instead of using tensors.Supports in-place operations.Prerequisitesnumpy(required): Only use its basic functions.scipy(optional): Can be installed to improve efficiency in some functions:scipy.special.erfused innn.GELU. approximation of erf is used when scipy is not installed.scipy.sparseused innn.Embeddingwhen settingsparse=True. Can't use sparse if neither scipy or cupy is installed.cupy(optional): Compute Numpy functions in GPU. Use.cuda()/.cpu()to send tensors or modules between CPU and GPU.pytorch(optional): Only use its dataset, dataloder and transforms, required in actual training to load and preprocess data. In each iteration, torch tensors (i.e. data, label) will be converted to tortto tensors bytortto_tensor = tortto.tensor(torch_tensor.numpy()).cuda(). All computations after that is done in tortto. See examples below.Quick startpytortto uses the same syntax as pytorch. Here shows the forward and backward propagation of some functions.importtorttoasttx=tt.tensor([[1.1,2.0,3.6],[4.7,3.14,2.718]],requires_grad=True)y=tt.tensor([[2.,3.99],[8.4,1.5],[2.5,7.8]],requires_grad=True)output=tt.tanh(1/(tt.cat([tt.sin((tt.exp(x**1.4)/3.1**tt.log(x))@y),y]).mean()))output.backward()print(output)# tensor(0.3489, grad_fn=<tanhBackward>)print(x.grad)# [[ -0.0276   0.6477 -11.827 ]#  [ 27.2554  -5.3062   1.0978]]print(y.grad)# [[-10.8005   8.2755]#  [ -0.3336   0.2187]#  [  0.8972  -0.9684]]in-place operations:importtorttoasttx0=tt.tensor([[-0.5722,-1.3110,-1.4200,-0.3545],[0.0945,-1.2108,-0.1059,0.8041],[-0.5110,1.4361,-1.1575,-0.7639]],requires_grad=True)x=x0+0# make it non-leaf for in-place operationsx[:,2:]=tt.sign(x[:,2:])*tt.sqrt(tt.abs_(x[:,2:]))x.backward(x)print(x._version)# 2print(x0.grad)# [[-0.5722 -1.3110 -0.5000 -0.5000]#  [ 0.0945 -1.2108 -0.5000  0.5000]#  [-0.5110  1.4361 -0.5000 -0.5000]]tortto also implements common Modules. Let's tryConv2dand compare its result and speed with pytorch:conv2d comparison in GPUquick start: reverse predictionNext, Let's train a transformer encoder in 40 seconds, inspired fromUvA Deep Learning Tutorials:The goal is to train a simple transformer encoder that can reverse the input sequence. i.e. if the input is4,1,9,7,5,2,2, the correct output would be2,2,5,7,9,1,4.ExamplesAll examples trained on a Tesla T4 (16GB memory) GPUWhen comparing speed with pytorch,torch.backends.cudnn.benchmarkis set toFalsefor a fair comparisonResnetTrained on CIFAR10Each epoch = 45k train + 5k validationTested on full 10k test samplesmodeltest acc.#.filtersnepochs trainedspeed (min/epoch)pytorch speed (min/epoch)speed comparisonsmall-preact_resnet11094.08%(16,32,64)basicBlock:(18,18,18)1803.651.23.65/1.2=3.0preact_resnet1894.65%(64,128,256,512)basicBlock:(2,2,2,2)1802.00.752.0/0.75=2.7standard_resnet50 (finetune)96.31%(64,128,256,512)bottleNeck:(3,4,6,3)1520.658.8220.65/8.82=2.3preact_resnet10194.78%(64,128,256,512)bottleNeck:(3,4,23,3)2009.24.39.2/4.3=2.1small_preact_resnet110,preact_resnet18andpreact_resnet101are trained from scratch. Kernel size of the first conv layer is 3, because CIFAR-10 images are 32x32 in size.standard_resnet50fintunes the full pretrained model. Kernel size of the first conv layer is 7. CIFAR-10 images are resized to 224x224 before feeding into the model.UNetTrained on the carvana datasetGPU memory: 1.5GBTraining time: 7 minsmodelimage size#.filtersbatchsizetrain/val/testepochsUNetresized to 3x64x6432,64,128,256,512,256,128,64,32323658/646/78420Vision TransformerTrained on CIFAR-10Finetune: Each epoch = 4500 train + 500 validationTested on full 10k test samplesmodeltest acc.layersHidden sizeMLP sizeHeadsepochs trainedspeed (min/epoch)pytorch speed (min/epoch)speed comparisonViT-B/16 (finetune)97.42%12768307212154.22.74.2/2.7=1.56ViT-B/16Finetunes the full pretrained model. CIFAR-10 images are resized to 224x224 before feeding into the model.DCGANDataset adapted fromhttps://github.com/bchao1/Anime-Face-DatasetImages resized to 3x64x64modelepochs trainedspeed (min/epoch)pytorch speed (min/epoch)speed comparisonDCGAN1006.272.386.27/2.38=2.6YOLOv1YOLO v1is work in progress...