Understanding torch.utils.bottleneck results

rendres: <p>I just finished running the bottleneck on my script and I’m pretty lost with one section. There is a section called autograd profiler output (CUDA mode) and another called autograd profiler output (CPU mode). The top events are IndexBackward and <em>index_put_impl</em>. I’m not sure what those refer to but most importantly are the times that it lists for these.</p>
<p>In CPU mode it shows this:</p>
<p>Self CPU time total: 341.996s<br>
CUDA time total: 0.000us</p>
<p>In CUDA mode it shows this:</p>
<p>Self CPU time total: 355.418s<br>
CUDA time total: 15.872ms</p>
<p>What exactly are these times. Should I be concerned that in CUDA mode there is still so much time on the CPU.</p>
<p>This model runs very slowly with very high memory use and I’m only using a batch size of 16. Any more and it crashes.</p>
<p>Any help would be greatly appreciated.</p>

ptrblck: <p>This might indicate that you are running into a CPU bottleneck, e.g. from data loading.<br>
You could try to adapt the data loading measurements from the <a href="https://github.com/pytorch/examples/blob/b9f3b2ebb9464959bdbf0c3ac77124a704954828/imagenet/main.py#L280">ImageNet example</a> to check, if that’s the case or not.</p>

rendres: <p>So I played around with the timers in that code (which is awesome by the way, thank you) and also did a few little experiments with torch.save and torch.load. I tried to be slick and save the part which took the longest each time so that I could just pull it back up and train. Yeah, it doesn’t work that way.</p>
<p>What I did learn though was that my code takes the longest time when I do the following:</p>
<p>loss_dict = model(imgs, annotations)</p>
<p>When that part get’s executed it takes almost all my memory and things start to slow down. Maybe I should mention that I have a Nvidia Xavier. It have the 32GB model with a 512GB drive attached (this is where I store all my stuff). My dataset is about 8600 images (512 x 640). It’s a COCO dataset and I’m using a Faster RCNN - Resnet50.</p>
<p>It takes me about 30 hours to train this model. Perhaps that’s just what to expect from my gear but looking at others examples, somehow it just seems like this is slow. If it is slow then I’ve definitely done something wrong in my code.</p>
<p>Am I just expecting more than I should?</p>

ptrblck: <p>Which Xavier platform are you using? Xavier are embedded devices, which will not yield the same training performance as a server GPU.</p>

rendres: <p>I got the Xavier AGX Developer kit.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <a href="https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-agx-xavier/" target="_blank" rel="nofollow noopener">NVIDIA</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-agx-xavier/" target="_blank" rel="nofollow noopener">Deploy AI-Powered Autonomous Machines at Scale</a></h3>

<p>NVIDIAJetson AGX Xavier sets a new bar for compute density, energy efficiency, and AI inferencing capabilities on edge devices.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Looking at it now, this little guy will work wonders for my edge application when doing inference but when I look at specs on a server size GPU, it makes sense it would be slower when it comes to training.</p>
<p>I’ll have to make do with the times I’m seeing.</p>