Volatile GPU util 0% with high memory usage

Fahad_Jahangir: <p>I have found this question posted on this forum for the multiple times with no working answer.</p>
<p>I am trying to train a SegNet on satellite images using single GPU (Nvidia Tesla-k80 12GB). Memory-Usage is high but the volatile GPU-Util is 0%.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/00c7304cd71a2aabd0d074de2ed259b49ce7272e.png" data-download-href="https://discuss.pytorch.org/uploads/default/00c7304cd71a2aabd0d074de2ed259b49ce7272e" title="prob"><img src="https://discuss.pytorch.org/uploads/default/original/2X/0/00c7304cd71a2aabd0d074de2ed259b49ce7272e.png" alt="prob" data-base62-sha1="6SL6wHEd4i9vf6Cu0Yj4e9Z3Ii" width="690" height="424" data-dominant-color="3C1932"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">prob</span><span class="informations">717×441 39.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
In the DataLoader, I have tried increasing the <strong>num_workers</strong>, setting the <strong>pin_memory= True</strong>, and <strong>removed all the preprocessing</strong> like Data Augmentation, Caching etc… but the problem is still persistent. The code works just fine if I change the dataset.</p>
<p>Any help will be highly appreciated.</p>

ptrblck: <aside class="quote no-group" data-username="Fahad_Jahangir" data-post="1" data-topic="52047">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fahad_jahangir/48/14587_2.png" class="avatar"> Fahad_Jahangir:</div>
<blockquote>
<p>The code works just fine if I change the dataset.</p>
</blockquote>
</aside>
<p>What did you change regarding the dataset?<br>
If the samples are smaller or you can preload the other dataset completely, it points to a data loading bottleneck using your current dataset.</p>

Aviv_Shamsian: <p>Hi Patrick,<br>
What are the general guidelines in order to make optimal GPU and CPU usage. I faced the same problem mentioned here when I tried to run the 'training classifier" notebook tutorial. When I changed the batch size nothing changed in the GPU usage (~600-700MB).<br>
The setup:<br>
CIFAR10 dataset<br>
2 X 1080 ti (nvidia)<br>
8 processors 32GB RAM memory</p>

Fahad_Jahangir: <p>Yes, the other dataset have smaller samples (images with low spatial resolution).<br>
How can I avoid this bottleneck situation?</p>

ptrblck: <p><a class="mention" href="/u/rwightman">@rwightman</a> shared some insights on data bottlenecks <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">here</a>.<br>
CC <a class="mention" href="/u/aviv_shamsian">@Aviv_Shamsian</a></p>

Aviv_Shamsian: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thank you for your quick response!<br>
Can you please address to my specific problem? I read the comment you tagged me in but I have done all is written there. I’m using the Pytorch Dataloader (without any unnecessary for loops). How can I be sure what is the main cause of this problem? When I use <code>htop</code> in terminal all the six processors assigned to the mission are loaded with work but the GPU stays in ~600-700MB capacity regardless to the batch size (which increased to 2048 at the most). This is the reason why I suspect that the preprocess part is the bottleneck in this example.  Just to mention no augmentations where preformed only reading, normalizing and converting to tensor.</p>

Fahad_Jahangir: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for you response. I realized that the sample size was very large and took so long to load, So I tried cashing the data, it takes more memory which I can afford however once the data was cached, Volatile GPU-Util reached maximum.</p>