GPU utilization reduces on large dataset

nishadsinghi: <p>I am trying to train a CLIP model on a cluster at my institute using 4 A100 GPUs. When I try to train it on a dataset of 3M images, the cluster shows that only 2 GPUs are being properly utilized (screenshot). On the other hand, when I try to train it on a subset containing only 500k images, all 4 GPUs get utilized. Everything else is the same, I just change the location of the training data. I am not sure why this is happening, and how I can even begin to diagnose this issue. What can I do to get the most out of my GPUs?<br>
The model reads a csv file containing the locations of training images and their corresponding captions, and the training data is stored on a hard drive on the cluster.</p>
<p>Thanks!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/fff83c3aab26fcfcc585e1e41405663f708d3fdd.png" data-download-href="https://discuss.pytorch.org/uploads/default/fff83c3aab26fcfcc585e1e41405663f708d3fdd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/fff83c3aab26fcfcc585e1e41405663f708d3fdd_2_345x499.png" alt="image" data-base62-sha1="AwpAZfgAnkM07SjFZ58ebm0wPxj" width="345" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/fff83c3aab26fcfcc585e1e41405663f708d3fdd_2_345x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/fff83c3aab26fcfcc585e1e41405663f708d3fdd_2_517x748.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/f/fff83c3aab26fcfcc585e1e41405663f708d3fdd_2_690x998.png 2x" data-dominant-color="EAD0C7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">851×1231 124 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

J_Johnson: <p>Are you using DP or DDP?</p>
<p><a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/ddp_tutorial.html</a></p>

nishadsinghi: <p>The implementation I’m trying out (<a href="https://github.com/goel-shashank/CyCLIP" class="inline-onebox" rel="noopener nofollow ugc">GitHub - goel-shashank/CyCLIP</a>) uses DDP.</p>

knoriy: <p>This sounds like an I/O issue. Are you using an iterable dataset? If not, random reading and writing on a spinning hard drive are extremely slow. Therefore, your GPUs are waiting for the data.</p>
<p>You can solve this by sharding your datasets and reading the data sequentially.</p>

nishadsinghi: <p>Thanks for your reply. I would like to make minimal changes to the implementation provided in the Github repository, so I’ll keep sharding as a last resort. Regarding your first point, I don’t understand why this issue doesn’t happen for 500k samples and only for 3M samples.</p>

knoriy: <p>It’s hard to say exactly why this happens without knowing more about the dataset, the dataset pipeline and the setup.</p>
<p>Regarding your question, reading across a large section of your drive can be a possible cause, especially if you are making random reads and writing calls.</p>
<p>You could use tools like <code>viztracer</code> to find the bottleneck, This should give you a better idea of what could be causing it and point you in the right direction. Another common mistake is not setting <code>num_workers</code> to load and parse the data.</p>
<p>If the machine has a faster drive (such as an NVME drive), you could try moving your dataset onto it.</p>
<p>TLDR in order of ease:</p>
<ul>
<li>Try using large <code>num_workers</code>
</li>
<li>Move the dataset to a faster drive NVME or SATA SDD if possible</li>
<li>use <code>viztracer</code> or similar tool to visualise time-consuming task</li>
</ul>