Training limited to 2 CPUs

johnorford: <p>While training, I generally only use 2 CPUs at 100% utilisation (8 available). GPU utilisation is very low.</p>
<p>During validation, all CPUs are mostly used, and GPU utilisation is much higher.</p>
<p>During training, I:</p>
<ul>
<li>
<p>set workers to 1 (only one CPU is used), 2 (2 CPUs are used) and 3 or more (mostly, no more than 2 CPUs are used both 100%)</p>
</li>
<li>
<p>printed <code>torch.utils.data.get_worker_info()</code>. For example, I see the 8 worker IDs s expected. But they seem to be just on two CPUs</p>
</li>
</ul>
<p>I suspect I am missing something obvious…</p>
<p>There must be some difference in <code>__get_item__</code> (same function, but training of course does some augmentation…)…</p>
<p>Training:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/1/010a673eb7dfe6342d9709b2bf7b476bcd2e2dbd.png" data-download-href="https://discuss.pytorch.org/uploads/default/010a673eb7dfe6342d9709b2bf7b476bcd2e2dbd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/010a673eb7dfe6342d9709b2bf7b476bcd2e2dbd_2_690x329.png" alt="image" data-base62-sha1="9cLwbZSmvPbMaAigjJTewJOFMV" width="690" height="329" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/010a673eb7dfe6342d9709b2bf7b476bcd2e2dbd_2_690x329.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/010a673eb7dfe6342d9709b2bf7b476bcd2e2dbd_2_1035x493.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/1/010a673eb7dfe6342d9709b2bf7b476bcd2e2dbd_2_1380x658.png 2x" data-dominant-color="E2E9E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1432×683 178 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Validation:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d77b76f817e58bca734ca84533d4fa1e5b433bfb.png" data-download-href="https://discuss.pytorch.org/uploads/default/d77b76f817e58bca734ca84533d4fa1e5b433bfb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d77b76f817e58bca734ca84533d4fa1e5b433bfb_2_690x329.png" alt="image" data-base62-sha1="uKfb2BdTlDvlbmZM4cwcAytVD2r" width="690" height="329" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d77b76f817e58bca734ca84533d4fa1e5b433bfb_2_690x329.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d77b76f817e58bca734ca84533d4fa1e5b433bfb_2_1035x493.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d77b76f817e58bca734ca84533d4fa1e5b433bfb_2_1380x658.png 2x" data-dominant-color="E2E9E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1423×680 181 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/johnorford">@johnorford</a>,</p>
<p>Have you used the <code>torch.set_num_threads</code> function to change the number of threads used for intraop-parrallelism? A bit of advice, don’t max out the number of threads being used, it’s best to leave 1 or 2 free. Docs here: <a href="https://pytorch.org/docs/stable/generated/torch.set_num_threads.html" class="inline-onebox" rel="noopener nofollow ugc">torch.set_num_threads — PyTorch 2.0 documentation</a></p>
<p>You can see how many threads you’re using at the moment with <code>torch.get_num_threads</code> for reference. Docs here: <a href="https://pytorch.org/docs/stable/generated/torch.get_num_threads.html" class="inline-onebox" rel="noopener nofollow ugc">torch.get_num_threads — PyTorch 2.0 documentation</a></p>
<p>There’s also some docs on optimizing the number of threads here too: <a href="https://pytorch.org/docs/stable/notes/cpu_threading_torchscript_inference.html" class="inline-onebox" rel="noopener nofollow ugc">CPU threading and TorchScript inference — PyTorch 2.0 documentation</a></p>

johnorford: <p>Thanks.</p>
<p>By default <code>num_threads</code> for me was 4. I set it to 8 and 16 and did not see any difference.</p>
<p>(I also double checked, by printing the num_threads…)</p>

johnorford: <p>After some more investigation.</p>
<p>It seems to me that the data preparation part of the the data loader worker is just being run on two cores.</p>
<p>More details below…</p>
<p>I checked where each process is running.</p>
<p>Both validation &amp; training:</p>
<ul>
<li>6 workers</li>
<li>6 processes (always running on cores 2 &amp; 5) with 2 children</li>
<li>6 processes evenly utilise the two cores (33% each)</li>
<li>When data is loaded into GPU, then I see other cores activated (more obvious during validation)</li>
</ul>
<p>Training:</p>
<ul>
<li>Lower GPU %, sending to GPU less often</li>
<li>I assume the two cores that are activated are loading data prior to sending to GPU</li>
</ul>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6197e6feb54fe41a0020707497a398ccb51a09c.png" data-download-href="https://discuss.pytorch.org/uploads/default/b6197e6feb54fe41a0020707497a398ccb51a09c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6197e6feb54fe41a0020707497a398ccb51a09c_2_337x500.png" alt="image" data-base62-sha1="pYVwf2bFOZ3SpzQuX4BuLBiyAAA" width="337" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6197e6feb54fe41a0020707497a398ccb51a09c_2_337x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6197e6feb54fe41a0020707497a398ccb51a09c_2_505x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6197e6feb54fe41a0020707497a398ccb51a09c_2_674x1000.png 2x" data-dominant-color="E5EFE8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">718×1064 89.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Validation:</p>
<ul>
<li>Loads data into GPU more often</li>
<li>when data is loaded into GPU (i.e. you can see GPU utilisation also) then remaining cores are activated</li>
</ul>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d33cdb07a40b6526f5a6ef85b2f576dda55e842.png" data-download-href="https://discuss.pytorch.org/uploads/default/4d33cdb07a40b6526f5a6ef85b2f576dda55e842" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d33cdb07a40b6526f5a6ef85b2f576dda55e842_2_381x500.png" alt="image" data-base62-sha1="b0XKJ3jdUFrrwWJwQOuiGFgMHzY" width="381" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d33cdb07a40b6526f5a6ef85b2f576dda55e842_2_381x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d33cdb07a40b6526f5a6ef85b2f576dda55e842_2_571x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d33cdb07a40b6526f5a6ef85b2f576dda55e842_2_762x1000.png 2x" data-dominant-color="E2EDE0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">786×1031 221 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

johnorford: <p>I ended up wrapping the model in DistributedDataParallel:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/c/3c0e01ae895502770416b6b9963b30879e745362.png" alt="image" data-base62-sha1="8zgEK8uu9obUMIT5z9EjHY8ZjQC" width="586" height="70"></p>
<p>An epoch that took 4 minutes is now taking about 3 minutes.</p>
<p>Not bad. Probably still not optimal.</p>
<p>The DDP module</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9b46bce594061403875df15e0cf3335a0d2dbdb0.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9b46bce594061403875df15e0cf3335a0d2dbdb0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9b46bce594061403875df15e0cf3335a0d2dbdb0_2_627x500.jpeg" alt="image" data-base62-sha1="m9DyUEjSLmhhL86dsfd2j3ZzxgQ" width="627" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9b46bce594061403875df15e0cf3335a0d2dbdb0_2_627x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9b46bce594061403875df15e0cf3335a0d2dbdb0_2_940x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/b/9b46bce594061403875df15e0cf3335a0d2dbdb0_2_1254x1000.jpeg 2x" data-dominant-color="DAEDE0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1494×1190 187 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It looks like the default data loader uses some more complicated way of distributing things, but DDP looks much simpler (compare the HTop outputs).</p>
<p>No more bottlenecks with two cores.</p>
<p>Suspect disk IO is the next bottleneck.</p>

johnorford: <p>Finally, I reduced the batch size by a lot, and now utilisation of CPUs and GPU is far high.</p>
<p>Epoch time is now down to ~1.5 minutes (from 4 minutes originally).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/1/618873249df90f8106667550a192a4f9c9dbf439.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/618873249df90f8106667550a192a4f9c9dbf439" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/618873249df90f8106667550a192a4f9c9dbf439_2_161x500.jpeg" alt="image" data-base62-sha1="dUOEJfENJRpPGDnhnOubFN7Stux" width="161" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/618873249df90f8106667550a192a4f9c9dbf439_2_161x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/618873249df90f8106667550a192a4f9c9dbf439_2_241x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/618873249df90f8106667550a192a4f9c9dbf439_2_322x1000.jpeg 2x" data-dominant-color="E3EFE6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">694×2147 344 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

RylanSchaeffer: <p><a class="mention" href="/u/johnorford">@johnorford</a> what program/command line utility are you displaying in these screenshots? It looks highly useful <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>