Package: torchtoolbox
Torch ToolboxStable Version: v0.1.5 (recommended)Automatic upload to PyPI is finished.This project aims to provide the most frequently used tools to help you write more concise and readable PyTorch code.Pull requests and issues are highly encouraged. Please reach out!Pytorch-ToolboxInstallingTodoUsageTools0. Now CV2 transforms have been released.1. Show your model parameters and FLOPs.2. Metric collection3. Model Initializer4. AdaptiveSequential5. Make and Use LMDB dataset.6. Non-Lable dataset7. Activation Layer8. FeatureVerification MetricFashion work1. LabelSmoothingLoss2. CosineWarmupLr3. SwitchNorm2d/3d4. Swish activation5. Lookahead optimizer5. Mixup training6. Cutout7. No decay bias8. Margin based classification loss9. DCNv210. FocalLoss11. SigmoidCrossEntropy12. CircleLoss13. EvoNrom14. Activation Layer15. Zero LastGamma Init16. SGD_GC17. AutoaugmentContributionInstallingAn easy way to install Torch Toolbox is via pip:pipinstalltorchtoolboxIf you want to install the nightly version:pipinstall-Ugit+https://github.com/PistonY/torch-toolbox.git@masterTodocv2 transforms.Integrate withalbumentationsPrepare tensorboard support with metric collection.UsageToolbox have two main parts:Auxiliary tools to make using Pytorch easier.Some fashion work which don't exist in Pytorch core.More examples could be found atModelZoo.pytorchTools0. Now CV2 transforms have been released.Support as list:(need test)__all__=["Compose","ToTensor","ToCVImage","Normalize","Resize","Scale","CenterCrop","Pad","Lambda","RandomApply","RandomChoice","RandomOrder","RandomCrop","RandomHorizontalFlip","RandomVerticalFlip","RandomResizedCrop","RandomSizedCrop","FiveCrop","TenCrop","LinearTransformation","ColorJitter","RandomRotation","RandomAffine","Grayscale","RandomGrayscale","RandomPerspective","RandomErasing","RandomGaussianNoise","RandomPoissonNoise","RandomSPNoise","RandomTransforms","Cutout"]Part of this code refers toopencv_transforms_torchvisionalbumentationsis also recommended as cv2 backend transform tools.1. Show your model parameters and FLOPs.importtorchfromtorchtoolbox.toolsimportsummaryfromtorchvision.models.mobilenetimportmobilenet_v2model=mobilenet_v2()summary(model,torch.rand((1,3,224,224)))Here are some short outputs.Layer (type)               Output Shape          Params    FLOPs(M+A) #
================================================================================
            Conv2d-1          [1, 64, 112, 112]            9408       235225088
       BatchNorm2d-2          [1, 64, 112, 112]             256         1605632
              ReLU-3          [1, 64, 112, 112]               0               0
         MaxPool2d-4            [1, 64, 56, 56]               0               0
          ...                      ...                      ...              ...
          Linear-158                  [1, 1000]         1281000         2560000
     MobileNetV2-159                  [1, 1000]               0               0
================================================================================
        Total parameters: 3,538,984  3.5M
    Trainable parameters: 3,504,872
Non-trainable parameters: 34,112
Total flops(M)  : 305,252,872  305.3M
Total flops(M+A): 610,505,744  610.5M
--------------------------------------------------------------------------------
Parameters size (MB): 13.502. Metric collectionWhen we train a model we usually need to calculate some metrics like accuracy(top1-acc), loss etc.
Now toolbox support the following:Accuracy: top-1 acc.TopKAccuracy: topK-acc.NumericalCost: This is a number metric collection which supportmean,max,mincalculate type.FeatureVerification.This is widely used in margin based algorithm.fromtorchtoolboximportmetric# define firsttop1_acc=metric.Accuracy(name='Top1 Accuracy')top5_acc=metric.TopKAccuracy(top=5,name='Top5 Accuracy')loss_record=metric.NumericalCost(name='Loss')# reset before usingtop1_acc.reset()top5_acc.reset()loss_record.reset()...model.eval()fordata,labelsinval_data:data=data.to(device,non_blocking=True)labels=labels.to(device,non_blocking=True)outputs=model(data)losses=Loss(outputs,labels)# update/recordtop1_acc.update(outputs,labels)top5_acc.update(outputs,labels)loss_record.update(losses)test_msg='Test Epoch{}:{}:{:.5},{}:{:.5},{}:{:.5}\n'.format(epoch,top1_acc.name,top1_acc.get(),top5_acc.name,top5_acc.get(),loss_record.name,loss_record.get())print(test_msg)Then you may get outputs like thisTest Epoch 101: Top1 Accuracy:0.7332, Top5 Accuracy:0.91514, Loss:1.06053. Model InitializerNow ToolBox supportXavierInitializerandKaimingInitializer.fromtorchtoolbox.nn.initimportKaimingInitializermodel=XXXinitializer=KaimingInitializer()model.apply(initializer)4. AdaptiveSequentialMake Pytorchnn.Sequentialcould handle multi input/output layer.fromtorchimportnnfromtorchtoolbox.nnimportAdaptiveSequentialimporttorchclassn_to_n(nn.Module):def__init__(self):super().__init__()self.conv1=nn.Conv2d(3,3,1,1,bias=False)self.conv2=nn.Conv2d(3,3,1,1,bias=False)defforward(self,x1,x2):y1=self.conv1(x1)y2=self.conv2(x2)returny1,y2classn_to_one(nn.Module):def__init__(self):super().__init__()self.conv1=nn.Conv2d(3,3,1,1,bias=False)self.conv2=nn.Conv2d(3,3,1,1,bias=False)defforward(self,x1,x2):y1=self.conv1(x1)y2=self.conv2(x2)returny1+y2classone_to_n(nn.Module):def__init__(self):super().__init__()self.conv1=nn.Conv2d(3,3,1,1,bias=False)self.conv2=nn.Conv2d(3,3,1,1,bias=False)defforward(self,x):y1=self.conv1(x)y2=self.conv2(x)returny1,y2seq=AdaptiveSequential(one_to_n(),n_to_n(),n_to_one()).cuda()td=torch.rand(1,3,32,32).cuda()out=seq(td)print(out.size())# output# torch.Size([1, 3, 32, 32])5. Make and Use LMDB dataset.If you meet IO speed limit, you may think aboutLMDBformat dataset.
LMDB is a tiny database with some excellent properties.Easy to generate a LMDB format dataset.fromtorchtoolbox.tools.convert_lmdbimportgenerate_lmdb_dataset,raw_readerfromtorchvision.datasetsimportImageFolderdt=ImageFolder(...,loader=raw_reader)save_dir=XXXdataset_name=YYYgenerate_lmdb_dataset(dt,save_dir=save_dir,name=dataset_name)Then if you useImageFolderlike dataset you can easily useImageLMDBto load you dataset.fromtorchtoolbox.dataimportImageLMDBdt=ImageLMDB(db_path=save_dir,db_name=dataset_name,...)6. Non-Lable datasetThis dataset only return images.More details please refers tocodes7. Activation LayerAActivationLayer is provided to select a activation by string.fromtorchtoolbox.nnimportActivationrelu=Activation('relu',auto_optimize=True)8. FeatureVerification MetricAFeatureVerification Metricused to test feature based accuracy.
More details refers to code.Fashion work1. LabelSmoothingLossfromtorchtoolbox.nnimportLabelSmoothingLoss# The num classes of your task should be defined.classes=10# LossLoss=LabelSmoothingLoss(classes,smoothing=0.1)...fori,(data,labels)inenumerate(train_data):data=data.to(device,non_blocking=True)labels=labels.to(device,non_blocking=True)optimizer.zero_grad()outputs=model(data)# just use as usual.loss=Loss(outputs,labels)loss.backward()optimizer.step()2. CosineWarmupLrCosine lr scheduler with warm-up epochs.It's helpful to improve acc for classification models.fromtorchtoolbox.optimizerimportCosineWarmupLroptimizer=optim.SGD(...)# define scheduler# `batches_pre_epoch` means how many batches(times update/step the model) within one epoch.# `warmup_epochs` means increase lr how many epochs to `base_lr`.# you can find more details in file.lr_scheduler=CosineWarmupLr(optimizer,batches_pre_epoch,epochs,base_lr=lr,warmup_epochs=warmup_epochs)...fori,(data,labels)inenumerate(train_data):...optimizer.step()# remember to step/update status here.lr_scheduler.step()...3. SwitchNorm2d/3dfromtorchtoolbox.nnimportSwitchNorm2d,SwitchNorm3dJust use it like Batchnorm2d/3d.
More details please refer to origin paperDifferentiable Learning-to-Normalize via Switchable NormalizationOpenSourse4. Swish activationfromtorchtoolbox.nnimportSwishJust use it like Relu.
More details please refer to origin paperSEARCHING FOR ACTIVATION FUNCTIONS5. Lookahead optimizerA wrapper optimizer seems better than Adam.Lookahead Optimizer: k steps forward, 1 step backfromtorchtoolbox.optimizerimportLookaheadfromtorchimportoptimoptimizer=optim.Adam(...)optimizer=Lookahead(optimizer)5. Mixup trainingMixup method to train a classification model.mixup: BEYOND EMPIRICAL RISK MINIMIZATIONfromtorchtoolbox.toolsimportmixup_data,mixup_criterion# set beta distributed parm, 0.2 is recommend.alpha=0.2fori,(data,labels)inenumerate(train_data):data=data.to(device,non_blocking=True)labels=labels.to(device,non_blocking=True)data,labels_a,labels_b,lam=mixup_data(data,labels,alpha)optimizer.zero_grad()outputs=model(data)loss=mixup_criterion(Loss,outputs,labels_a,labels_b,lam)loss.backward()optimizer.update()6. CutoutA image transform method.Improved Regularization of Convolutional Neural Networks with Cutoutfromtorchvisionimporttransformsfromtorchtoolbox.transformimportCutout_train_transform=transforms.Compose([transforms.RandomResizedCrop(224),Cutout(),transforms.RandomHorizontalFlip(),transforms.ColorJitter(0.4,0.4,0.4),transforms.ToTensor(),normalize,])7. No decay biasIf you train a model with big batch size, eg. 64k, you may need this,Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutesfromtorchtoolbox.toolsimportno_decay_biasfromtorchimportoptimmodel=XXXparameters=no_decay_bias(model)optimizer=optim.SGD(parameters,...)8. Margin based classification lossNow support:ArcLossCosLossL2Softmaxfromtorchtoolbox.nn.lossimportArcLoss,CosLoss,L2SoftmaxYou could use this likenn.CrossEntropyLoss9. DCNv2Deformable ConvNets v2: More Deformable, Better Results10. FocalLossFocal Loss for Dense Object Detection11. SigmoidCrossEntropy12. CircleLossCircle Loss: A Unified Perspective of Pair Similarity Optimization13. EvoNromEvolving Normalization-Activation Layers14. Activation LayerSwishHardSwishHardSigmoid
Usage:fromtorchtoolbox.nnimportSwish,HardSwish,HardSigmoidswish=Swish()hswish=HardSwish()hsigmoid=HardSigmoid()15. Zero LastGamma Initfromtorchtoolbox.nn.initimportZeroLastGammamodel==XXXinit=ZeroLastGamma(block_name='Bottleneck',bn_name='bn3')model.apply(init)16. SGC_GCGradient Centralization: A New Optimization Technique for Deep Neural Networksfromtorchtoolbox.optimizerimportSGD_GCoptimizer=SGD_GC(model.parameters())optimizer.zero_grad()loss.backward()optimizer.step()17. Autoaugment# for ImageNetfromtorchtoolbox.transformimportImageNetPolicy,Compose,\RandomResizedCrop,RandomHorizontalFlip,ToTensortransforms=Compose([RandomResizedCrop(224),RandomHorizontalFlip(),ImageNetPolicy,ToTensor()])ContributionWelcome pull requests and issues!!!