How to speedup the back-propagation function of AdderNet

username: promach
cooked: <p>How to speedup the <a href="https://github.com/promach/PyTorch-YOLOv3/blob/addernet/adder.py#L67=L100" rel="nofollow noopener">back-propagation function of AdderNet</a> because training time is extremely long ?</p>
<p>Note: More context could be found at <a href="https://github.com/huawei-noah/AdderNet/issues/16#issuecomment-627446321" rel="nofollow noopener">https://github.com/huawei-noah/AdderNet/issues/16#issuecomment-627446321</a></p>

username: albanD
cooked: <p>What is slow in there? The cuda kernels?</p>

username: promach
cooked: <p>I tested the <a href="https://colab.research.google.com/drive/1V94-WC6Jf8M4Tj8iX3x5CuW93qSfuG6A#scrollTo=v-9VE8CK1vPy" rel="nofollow noopener">cuda kernels</a> separately on its own, and it is not slow compared to the cuda kernel(<a href="https://github.com/promach/PyTorch-YOLOv3/blob/addernet/unoptimized_cuda.cpp" rel="nofollow noopener">unoptimized_cuda.cpp</a>) used <a href="https://colab.research.google.com/drive/1VCafwykgNKAO6144LssBFFy0TmruDNSE#scrollTo=UD_HuiOq9nBW" rel="nofollow noopener">together with YOLO</a></p>
<p>Why ?</p>
<pre><code class="lang-auto">Streaming output truncated to the last 5000 lines.
17.9488s  1.6960us              (1 1 1)        (64 1 1)         8        0B        0B         -           -           -           -     Tesla P4 (0)         1         7  _ZN2at6native6modern29vectorized_elementwise_kernelILi4EZZZNS0_16fill_kernel_cudaERNS_14TensorIteratorEN3c106ScalarEENKUlvE_clEvENKUlvE2_clEvEUlvE_NS_6detail5ArrayIPcLi1EEEEEviT0_T1_ [6598]
17.9488s  3.5840us              (1 1 1)       (256 1 1)        48        0B        0B         -           -           -           -     Tesla P4 (0)         1         7  _ZN89_GLOBAL__N__65_tmpxft_00002210_00000000_11_DistributionNormal_compute_75_cpp1_ii_ccb2fd7d43distribution_elementwise_grid_stride_kernelIfLi4EZZZN2at6native18normal_kernel_cudaERNS1_14TensorIteratorEddPNS1_9GeneratorEENKUlvE_clEvENKUlvE0_clEvEUlP24curandStatePhilox4_32_10E0_ZNS_27distribution_nullary_kernelIffLi4ENS1_13CUDAGeneratorESB_ZZZNS2_18normal_kernel_cudaES4_ddS6_ENKS7_clEvENKS8_clEvEUlfE_EEvS4_PT2_RKT3_T4_EUlifE_EEviSt4pairImmET1_SF_ [6606]
17.9489s  1.5680us              (1 1 1)        (64 1 1)         8        0B        0B         -           -           -           -     Tesla P4 (0)         1         7  _ZN2at6native6modern29vectorized_elementwise_kernelILi4EZZZNS0_16fill_kernel_cudaERNS_14TensorIteratorEN3c106ScalarEENKUlvE_clEvENKUlvE2_clEvEUlvE_NS_6detail5ArrayIPcLi1EEEEEviT0_T1_ [6611]
17.9489s  3.6800us              (1 1 1)       (256 1 1)        48        0B        0B         -           -           -           -     Tesla P4 (0)         1         7  _ZN89_GLOBAL__N__65_tmpxft_00002210_00000000_11_DistributionNormal_compute_75_cpp1_ii_ccb2fd7d43distribution_elementwise_grid_stride_kernelIfLi4EZZZN2at6native18normal_kernel_cudaERNS1_14TensorIteratorEddPNS1_9GeneratorEENKUlvE_clEvENKUlvE0_clEvEUlP24curandStatePhilox4_32_10E0_ZNS_27distribution_nullary_kernelIffLi4ENS1_13CUDAGeneratorESB_ZZZNS2_18normal_kernel_cudaES4_ddS6_ENKS7_clEvENKS8_clEvEUlfE_EEvS4_PT2_RKT3_T4_EUlifE_EEviSt4pairImmET1_SF_ [6619]
17.9489s  1.6960us              (1 1 1)        (64 1 1)         8        0B        0B         -           -           -           -     Tesla P4 (0)         1         7  _ZN2at6native6modern29vectorized_elementwise_kernelILi4EZZZNS0_16fill_kernel_cudaERNS_14TensorIteratorEN3c106ScalarEENKUlvE_clEvENKUlvE2_clEvEUlvE_NS_6detail5ArrayIPcLi1EEEEEviT0_T1_ [6624]
17.9490s  3.6480us              (1 1 1)       (256 1 1)        48        0B        0B         -           -           -           -     Tesla P4 (0)         1         7  _ZN89_GLOBAL__N__65_tmpxft_00002210_00000000_11_DistributionNormal_compute_75_cpp1_ii_ccb2fd7d43distribution_elementwise_grid_stride_kernelIfLi4EZZZN2at6native18normal_kernel_cudaERNS1_14TensorIteratorEddPNS1_9GeneratorEENKUlvE_clEvENKUlvE0_clEvEUlP24curandStatePhilox4_32_10E0_ZNS_27distribution_nullary_kernelIffLi4ENS1_13CUDAGeneratorESB_ZZZNS2_18normal_kernel_cudaES4_ddS6_ENKS7_clEvENKS8_clEvEUlfE_EEvS4_PT2_RKT3_T4_EUlifE_EEviSt4pairImmET1_SF_ [6632]
17.9490s  1.5680us              (1 1 1)        (64 1 1)         8        0B        0B         -           -           -           -     Tesla P4 (0)         1         7  _ZN2at6native6modern29vectorized_elementwise_kernelILi4EZZZNS0_16fill_kernel_cudaERNS_14TensorIteratorEN3c106ScalarEENKUlvE_clEvENKUlvE2_clEvEUlvE_NS_6detail5ArrayIPcLi1EEEEEviT0_T1_ [6637]
17.9491s  3.6480us              (1 1 1)       (256 1 1)        48        0B        0B         -           -           -           -     Tesla P4 (0)         1         7  _ZN89_GLOBAL__N__65_tmpxft_00002210_00000000_11_DistributionNormal_compute_75_cpp1_ii_ccb2fd7d43distribution_elementwise_grid_stride_kernelIfLi4EZZZN2at6native18normal_kernel_cudaERNS1_14TensorIteratorEddPNS1_9GeneratorEENKUlvE_clEvENKUlvE0_clEvEUlP24curandStatePhilox4_32_10E0_ZNS_27distribution_nullary_kernelIffLi4ENS1_13CUDAGeneratorESB_ZZZNS2_18normal_kernel_cudaES4_ddS6_ENKS7_clEvENKS8_clEvEUlfE_EEvS4_PT2_RKT3_T4_EUlifE_EEviSt4pairImmET1_SF_ [6645]
17.9491s  1.7280us              (1 1 1)        (64 1 1)         8        0B        0B         -           -           -           -     Tesla P4 (0)         1         7  _ZN2at6native6modern29vectorized_elementwise_kernelILi4EZZZNS0_16fill_kernel_cudaERNS_14TensorIteratorEN3c106ScalarEENKUlvE_clEvENKUlvE2_clEvEUlvE_NS_6detail5ArrayIPcLi1EEEEEviT0_T1_ [6650]
17.9492s  3.6480us              (1 1 1)       (256 1 1)        48        0B        0B         -           -           -           -     Tesla P4 (0)         1         7  _ZN89_GLOBAL__N__65_tmpxft_00002210_00000000_11_DistributionNormal_compute_75_cpp1_ii_ccb2fd7d43distribution_elementwise_grid_stride_kernelIfLi4EZZZN2at6native18normal_kernel_cudaERNS1_14TensorIteratorEddPNS1_9GeneratorEENKUlvE_clEvENKUlvE0_clEvEUlP24curandStatePhilox4_32_10E0_ZNS_27distribution_nullary_kernelIffLi4ENS1_13CUDAGeneratorESB_ZZZNS2_18normal_kernel_cudaES4_ddS6_ENKS7_clEvENKS8_clEvEUlfE_EEvS4_PT2_RKT3_T4_EUlifE_EEviSt4pairImmET1_SF_ [6658]
17.9492s  1.5680us              (1 1 1)        (64 1 1)         8        0B        0B         -           -           -           -     Tesla P4 (0)         1         7  _ZN2at6native6modern29vectorized_elementwise_kernelILi4EZZZNS0_16fill_kernel_cudaERNS_14TensorIteratorEN3c106ScalarEENKUlvE_clEvENKUlvE2_clEvEUlvE_NS_6detail5ArrayIPcLi1EEEEEviT0_T1_ [6663]
17.9492s  3.5840us              (1 1 1)       (256 1 1)        48        0B        0B         -           -           -           -     Tesla P4 (0)         1         7  _ZN89_GLOBAL__N__65_tmpxft_00002210_00000000_11_DistributionNormal_compute_75_cpp1_ii_ccb2fd7d43distribution_elementwise_grid_stride_kernelIfLi4EZZZN2at6native18normal_kernel_cudaERNS1_14TensorIteratorEddPNS1_9GeneratorEENKUlvE_clEvENKUlvE0_clEvEUlP24curandStatePhilox4_32_10E0_ZNS_27distribution_nullary_kernelIffLi4ENS1_13CUDAGeneratorESB_ZZZNS2_18normal_kernel_cudaES4_ddS6_ENKS7_clEvENKS8_clEvEUlfE_EEvS4_PT2_RKT3_T4_EUlifE_EEviSt4pairImmET1_SF_ [6671]
17.9492s  1.7280us              (1 1 1)        (64 1 1)         8        0B        0B         -           -           -           -     Tesla P4 (0)         1         7  _ZN2at6native6modern29vectorized_elementwise_kernelILi4EZZZNS0_16fill_kernel_cudaERNS_14TensorIteratorEN3c106ScalarEENKUlvE_clEvENKUlvE2_clEvEUlvE_NS_6detail5ArrayIPcLi1EEEEEviT0_T1_ [6676]
17.9493s  3.6480us              (1 1 1)       (256 1 1)        48        0B        0B         -           -           -           -     Tesla P4 (0)         1         7  _ZN89_GLOBAL__N__65_tmpxft_00002210_00000000_11_DistributionNormal_compute_75_cpp1_ii_ccb2fd7d43distribution_elementwise_grid_stride_kernelIfLi4EZZZN2at6native18normal_kernel_cudaERNS1_14TensorIteratorEddPNS1_9GeneratorEENKUlvE_clEvENKUlvE0_clEvEUlP24curandStatePhilox4_32_10E0_ZNS_27distribution_nullary_kernelIffLi4ENS1_13CUDAGeneratorESB_ZZZNS2_18normal_kernel_cudaES4_ddS6_ENKS7_clEvENKS8_clEvEUlfE_EEvS4_PT2_RKT3_T4_EUlifE_EEviSt4pairImmET1_SF_ [6684]
17.9493s  1.6000us              (1 1 1)        (64 1 1)         8        0B        0B         -           -           -           -     Tesla P4 (0)         1         7  _ZN2at6native6modern29vectorized_elementwise_kernelILi4EZZZNS0_16fill_kernel_cudaERNS_14TensorIteratorEN3c106ScalarEENKUlvE_clEvENKUlvE2_clEvEUlvE_NS_6detail5ArrayIPcLi1EEEEEviT0_T1_ [6689]
17.9493s  3.6800us              (1 1 1)       (256 1 1)        48        0B        0B         -           -           -           -     Tesla P4 (0)         1         7  _ZN89_GLOBAL__N__65_tmpxft_00002210_00000000_11_DistributionNormal_compute_75_cpp1_ii_ccb2fd7d43distribution_elementwise_grid_stride_kernelIfLi4EZZZN2at6native18normal_kernel_cudaERNS1_14TensorIteratorEddPNS1_9GeneratorEENKUlvE_clEvENKUlvE0_clEvEUlP24curandStatePhilox4_32_10E0_ZNS_27distribution_nullary_kernelIffLi4ENS1_13CUDAGeneratorESB_ZZZNS2_18normal_kernel_cudaES4_ddS6_ENKS7_clEvENKS8_clEvEUlfE_EEvS4_PT2_RKT3_T4_EUlifE_EEviSt4pairImmET1_SF_ [6697]
17.9494s  1.6960us              (1 1 1)        (64 1 1)         8        0B        0B         -           -           -           -     Tesla P4 (0)         1         7  _ZN2at6native6modern29vectorized_elementwise_kernelILi4EZZZNS0_16fill_kernel_cudaERNS_14TensorIteratorEN3c106ScalarEENKUlvE_clEvENKUlvE2_clEvEUlvE_NS_6detail5ArrayIPcLi1EEEEEviT0_T1_ [6702]
17.9494s  3.6480us              (1 1 1)       (256 1 1)        48        0B        0B         -           -           -           -     Tesla P4 (0)         1         7  
</code></pre>

username: albanD
cooked: <p>I’m a bit confused about what you’re doing here.<br>
What are these timings here for?</p>
<p>Also you mean that just using the kernels by themselves is faster than when using them embedding in the rest of the code?<br>
Maybe you’re not using the same input sizes? Or not testing the backward kernel?</p>

username: promach
cooked: <aside class="quote no-group" data-username="albanD" data-post="4" data-topic="84294">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>Also you mean that just using the kernels by themselves is faster than when using them embedding in the rest of the code?</p>
</blockquote>
</aside>
<p>Yes</p>
<p>But the CUDA kernel itself is only using CIFAR10, while YOLO together with CUDA kernel is using VOC2007</p>

username: albanD
cooked: <p>So these two have different input sizes right? Could it be that the kernel scales very badly with the input size?</p>

username: promach
cooked: <p>I am checking where exactly the CUDA kernel calculation <a href="https://github.com/jdnie/AdderNetCuda/blob/master/unoptimized/unoptimized.cu#L419-L554" rel="nofollow noopener">bottleneck</a> is</p>

username: promach
cooked: <aside class="quote no-group" data-username="albanD" data-post="6" data-topic="84294">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p>Could it be that the kernel scales very badly with the input size?</p>
</blockquote>
</aside>
<p>Do you know how to use VOC2007 dataset instead of the <a href="https://github.com/jdnie/AdderNetCuda/blob/master/train.py#L80" rel="noopener nofollow ugc">CIFAR-10</a> dataset (which is only of size 32 x 32) on the <a href="https://github.com/jdnie/AdderNetCuda/blob/master/train.py#L80" rel="noopener nofollow ugc">CUDA kernel itself</a> ?</p>
<p><a href="https://pytorch.org/docs/stable/torchvision/datasets.html" rel="noopener nofollow ugc">torchvision.dataset</a> does not have VOC2007</p>

username: promach
cooked: <pre><code class="lang-auto">    if args.dataset == 'cifar10':
        train_dataset = datasets.CIFAR10(
            root=args.data_dir,
            train=True,
            transform=transforms.Compose([
                transforms.RandomHorizontalFlip(),
                transforms.RandomCrop(size=32, padding=4),
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.4914, 0.4822, 0.4465],
                                     std=[0.2023, 0.1994, 0.2010]),
            ]), download=True)
        val_dataset = datasets.CIFAR10(
            root=args.data_dir,
            train=False, 
            transform=transforms.Compose([
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.4914, 0.4822, 0.4465],
                                     std=[0.2023, 0.1994, 0.2010]),
            ]), download=True)
    else:
        train_dataset = datasets.ImageFolder(
            root=args.data_dir, 
            transform=transforms.Compose([
                transforms.RandomHorizontalFlip(),
                transforms.Resize(224),
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                         std=[0.229, 0.224, 0.225])
            ]))
        val_dataset = datasets.ImageFolder(
            root=args.data_dir, 
            transform=transforms.Compose([
                transforms.Resize(224),
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                         std=[0.229, 0.224, 0.225])
            ]))  
</code></pre>
<p>what does the <code>transform</code> do ?</p>
<p>I mean the RandomCrop</p>
<p><a href="https://pytorch.org/docs/stable/torchvision/transforms.html#torchvision.transforms.RandomCrop" rel="nofollow noopener">transforms.RandomCrop</a> could not be applied to VOC2007 dataset which consists of images of different dimensions.</p>

username: ptrblck
cooked: <p>Do you have a particular question regarding the linked documentation for <code>RandomCrop</code>?</p>
<p>What do you mean by different dimensions and what is not working?<br>
Do you get any error message using this transformation?</p>

username: promach
cooked: <p>Now, I have the following draft code for loading VOC2007 dataset.</p>
<p>However, this is only for image itself, what about labels for classification purpose ?</p>
<p>Could anyone advise how to load both images and the corresponding labels correctly ?</p>
<pre><code class="lang-auto">        #os.system("mkdir -p /home/rog/Downloads/AdderNetCuda/dataset/voc2007")
        #os.system("cd /home/rog/Downloads/AdderNetCuda/dataset/voc2007")
        
        #os.system("wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar")
        #os.system("wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar")

        os.system("mkdir -p /home/rog/Downloads/AdderNetCuda/dataset/voc2007/test")
        os.system("mkdir -p /home/rog/Downloads/AdderNetCuda/dataset/voc2007/train")

        os.system("tar -C /home/rog/Downloads/AdderNetCuda/dataset/voc2007/test -xf VOCtest_06-Nov-2007.tar")
        os.system("tar -C /home/rog/Downloads/AdderNetCuda/dataset/voc2007/train -xf VOCtrainval_06-Nov-2007.tar")
    
        train_dataset = datasets.ImageFolder(
            root="/home/rog/Downloads/AdderNetCuda/dataset/voc2007/train/VOCdevkit/VOC2007/JPEGImages", #args.data_dir, 
            transform=transforms.Compose([
                transforms.RandomHorizontalFlip(),
                #transforms.Resize(224),
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                         std=[0.229, 0.224, 0.225])
            ]))
        val_dataset = datasets.ImageFolder(
            root="/home/rog/Downloads/AdderNetCuda/dataset/voc2007/test/VOCdevkit/VOC2007/JPEGImages", #args.data_dir, 
            transform=transforms.Compose([
                #transforms.Resize(224),
                transforms.ToTensor(),
                transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                         std=[0.229, 0.224, 0.225])
            ]))        

    train_loader = torch.utils.data.DataLoader(train_dataset,
        batch_size=args.batch_size, shuffle=False,num_workers=args.workers, pin_memory=True)
    val_loader = torch.utils.data.DataLoader(val_dataset,
        batch_size=args.batch_size, shuffle=False, num_workers=args.workers, pin_memory=True)
</code></pre>

username: ptrblck
cooked: <p>If you are dealing with target images and would like to apply the same random transformations on both the data and target, you could use the functional API as described in <a href="https://discuss.pytorch.org/t/torchvision-transfors-how-to-perform-identical-transform-on-both-image-and-target/10606/7">this post</a>.<br>
This would make sure to keep the correspondence between the data and target.</p>