1st Minibatch takes abnormally large amount of time - Nvidia Jetson AGX Orin

username: saisamarth
cooked: <p>Hi,</p>
<p>I have been trying to train/inference various vision models using Pytorch on the AGX Orin device. Across all 3 vision models (MobileNet, ResNet, Yolov8 (Ultralytics)), I have seen that the 1st minibatch always takes a very long time compared to the rest of the minibatches. Another thing I have noticed is the GPU utilization is 0 until the 1st minibatch is completed and only goes up from 2nd minibatch. Likewise, I notice a significant spike (then pretty stable) in power usage after the 1st minibatch. For example, in the case of MobileNet inferencing, the 1st minibatch takes ~2000 ms where as the rest of them take ~30 ms. I am using a Pytorch dataloader to feed images and using wrapper over jtop for logging minibatch times and other metrics. The Pytorch version I am using is 1.12, CUDA version 11.4 and jtop 4.1.2.</p>
<p>Does anyone know why this is the case?</p>

username: tom
cooked: <p>I would imagine that this is related to CUDA and PyTorch CUDA subsystem initialization which may involve compilation. One particular aspect might be whether you have a compute architecture that is not foreseen in the PyTorch build. As far as I understand, that causes CUDA to generate PTX on the fly. Others will know more.</p>
<p>Best regards</p>
<p>Thomas</p>

username: saisamarth
cooked: <p>I have built and installed the Nvidia recommended versions of PyTorch and torchvision for my Jetson’s JetPack version (<a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048" class="inline-onebox" rel="noopener nofollow ugc">PyTorch for Jetson - Announcements - NVIDIA Developer Forums</a>). I have the R34 (release), REVISION: 1.1 JetPack version and exact pytorch version is 1.12.0a0+2c916ef.nv22.3.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/c/fc7b6b83e0a7cf496c352e4f2b3b80c0a91ea830.png" data-download-href="https://discuss.pytorch.org/uploads/default/fc7b6b83e0a7cf496c352e4f2b3b80c0a91ea830" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc7b6b83e0a7cf496c352e4f2b3b80c0a91ea830_2_690x127.png" alt="image" data-base62-sha1="A1yKpMARhxgopzmvIxjT0MhA0QE" width="690" height="127" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc7b6b83e0a7cf496c352e4f2b3b80c0a91ea830_2_690x127.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/c/fc7b6b83e0a7cf496c352e4f2b3b80c0a91ea830_2_1035x190.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/c/fc7b6b83e0a7cf496c352e4f2b3b80c0a91ea830.png 2x" data-dominant-color="E6F8C8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1156×213 18.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So I think that GPU’s compute architecture is compatible with the PyTorch build and PyTorch does not need to generate PTX. But in any case, does generation of PTX take so long?</p>

username: tom
cooked: <p>Oh right, you’d be using the JetPack that should have it all figured out.<br>
Can you see what processes are taking the CPU during the wait?</p>

username: saisamarth
cooked: <p>There arn’t any other process except workload processes running. Also I not worried about CPU utilization… Not sure why GPU is held back until the first minibatch completed.</p>

username: ptrblck
cooked: <blockquote>
<p>For example, in the case of MobileNet inferencing, the 1st minibatch takes ~2000 ms where as the rest of them take ~30 ms.</p>
</blockquote>
<p>I don’t think ~2s to initialize PyTorch and CUDA and infer the first batch is abnormally long and would expect this warmup time.</p>

username: saisamarth
cooked: <p>But this is on MAXN mode on the Orin AGX. If we lower the number of cores or memory frequency, I see the first minibatch taking 15 seconds also.</p>

username: ptrblck
cooked: <p>Still, 2s is fast and I doubt you can lower it any further as it’s also not specific to Jetson devices.<br>
On an x86 workstation:</p>
<pre><code class="lang-python">time python -c "import torch; x = torch.randn(1)"

real	0m0.881s
user	0m1.590s
sys	0m1.665s


time python -c "import torch; x = torch.randn(1).cuda()"

real	0m0.976s
user	0m1.398s
sys	0m1.913s
</code></pre>