A100 training slower than V100

ashu1807: <p>I have moved my model from V100 to A100 and instead of seeing an increase in speed these has been a significant slowdown from 14.2 it/sec to 10.06 it/sec.<br>
cuda version 11.3<br>
Pytorch version 1.9.0+cu111<br>
I have been specifically using the code from GitHub repository (NLSPN)</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/zzangjinsun/NLSPN_ECCV20">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">

      <a href="https://github.com/zzangjinsun/NLSPN_ECCV20" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/daaaeb81ae0d58d8f3ce6ee733c8823474f4bb391dfd979c6ed2711ae46f0b1b/zzangjinsun/NLSPN_ECCV20" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/zzangjinsun/NLSPN_ECCV20" target="_blank" rel="noopener nofollow ugc">GitHub - zzangjinsun/NLSPN_ECCV20: Park et al., Non-Local Spatial Propagation...</a></h3>

  <p>Park et al., Non-Local Spatial Propagation Network for Depth Completion, ECCV, 2020 - GitHub - zzangjinsun/NLSPN_ECCV20: Park et al., Non-Local Spatial Propagation Network for Depth Completion, ECC...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
There is a apex dependency in the repository which I thought to be the issue, but removing it and using a single GPU training also suffers from this issue.</p>

ptrblck: <p>The repository depends on <a href="https://github.com/xvjiarui/Deformable-Convolution-V2-PyTorch">Deformable-Convolution-V2-PyTorch</a>, which seems to have been written ~3 years ago. Are you also seeing a slowdown without these custom layers or did you profile the model to see which operations are the bottleneck?</p>

ashu1807: <p>Even after removing these deformable convolution layer there is a speed drop from 16.31it/s to 11.99 it/s.</p>

ptrblck: <p>Thanks for the update! Could you post the code to initialize the model as well as the input shapes you are using, please?</p>

ashu1807: <pre><code class="lang-auto">dist.init_process_group(backend='nccl', init_method='env://',
                            world_size=args.num_gpus, rank=gpu)
    torch.cuda.set_device(gpu)

    # Prepare dataset
    data = get_data(args)

    data_train = data(args, 'train')
    data_val = data(args, 'val')

    sampler_train = DistributedSampler(
        data_train, num_replicas=args.num_gpus, rank=gpu)
    sampler_val = DistributedSampler(
        data_val, num_replicas=args.num_gpus, rank=gpu)

    batch_size = args.batch_size // args.num_gpus

    loader_train = DataLoader(
        dataset=data_train, batch_size=batch_size, shuffle=False,
        num_workers=args.num_threads, pin_memory=True, sampler=sampler_train,
        drop_last=True)
    loader_val = DataLoader(
        dataset=data_val, batch_size=1, shuffle=False,
        num_workers=args.num_threads, pin_memory=True, sampler=sampler_val,
        drop_last=False)

    # Network
    model = get_model(args)
    net = model(args)
    net.cuda(gpu)

    if gpu == 0:
        if args.pretrain is not None:
            assert os.path.exists(args.pretrain), \
                "file not found: {}".format(args.pretrain)

            checkpoint = torch.load(args.pretrain)
            net.load_state_dict(checkpoint['net'])

            print('Load network parameters from : {}'.format(args.pretrain))

    # Loss
    loss = get_loss(args)
    loss = loss(args)
    loss.cuda(gpu)

    # Optimizer
    optimizer, scheduler = utility.make_optimizer_scheduler(args, net)

    net = apex.parallel.convert_syncbn_model(net)
    net, optimizer = amp.initialize(net, optimizer, , opt_level=args.opt_level,
                                    verbosity=0)
   net = DDP(net)
   for epoch in range(1, args.epochs+1):
       for batch, sample in enumerate(loader_train):
            sample = {key: val.cuda(gpu) for key, val in sample.items()
                      if val is not None}

            if epoch == 1 and args.warm_up:
                warm_up_cnt += 1

                for param_group in optimizer.param_groups:
                    lr_warm_up = param_group['initial_lr'] \
                                 * warm_up_cnt / warm_up_max_cnt
                    param_group['lr'] = lr_warm_up

            optimizer.zero_grad()

            output = net(sample)

</code></pre>
<p>The code is taken from src/main.py from the NLSPN repository. There are two inputs Rgb torch.Size([24, 3, 224, 304]) and lidar torch.Size([24, 1, 224, 304]). I have even tried to remove the apex dependency. That does not cause any issues with respect to the slowdown time.</p>

ashu1807: <p>Any idea what could be issue?</p>

my3bikaht: <p>In the topic right below this one (<a href="https://discuss.pytorch.org/t/can-not-attain-better-performances-after-changing-nvidia-gpu/139222/8" class="inline-onebox">Can not attain better performances after changing nvidia GPU - #8 by tjk</a>) we found out that enabling cudnn benchmark can lead to a better perfomance on A5000.</p>

zimian_wei: <p>Did you find out what the issue is? I also encountered this problem.</p>

ashu1807: <p>No. <a class="mention" href="/u/my3bikaht">@my3bikaht</a> , it did not work. <a class="mention" href="/u/zimian_wei">@zimian_wei</a> , did not find any solution.</p>

zimian_wei: <p>I found that moving data to SSD helps.</p>