Running multiple independent models on single gpu on PyTorch windows11 (RTX 3090 TI)

jaiabhayk: <p>Hi there,<br>
Anyone having any experience of running independent multiple models on a single gpu (RTX 3090 or RTX 3090 TI on PyTorch + windows11 ). I tried running two models to train and it takes double time when i run two separate models in separate power shell on windows (is there any limitations on rtxt 3090 or rtx 3090 TI) ? [its cifar10 and RAM is 64 gb and gpu ram is 24 gb and dataset is very small to fit in the RAM, so no memory read/write bottleneck].</p>

ptrblck: <p>Check if other compute resources are already exhausted on the device, as it would also disallow the parallel execution of different workloads.<br>
If that’s not the case, check if Windows adds some limitation (I’m not using Windows and don’t know about all “modes” it could use the GPU in).</p>

jaiabhayk: <p><a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thanks, i checked that, i have 64 gb ram, and 24 gb gpu ram. And running cifar10 training which fits in the memory. When i run single model it takes around 4 minutes but when i run two independent in two seperately power shells, it takes 8 minutes for each which is weird and seems like some locking for RTX 3090 TI on windows11 ?</p>

ptrblck: <p>I wasn’t thinking about the device memory but the compute resources.</p>

J_Johnson: <ol>
<li>
<p>run cmd</p>
</li>
<li>
<p>type <code>nvidia-smi</code></p>
</li>
<li>
<p>screenshot</p>
</li>
</ol>
<p>This will show what percent of your device CUDA cores are being utilized.</p>

jaiabhayk: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6aebf4112a2dafc7ad73b990200da9139a74fcb6.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/6aebf4112a2dafc7ad73b990200da9139a74fcb6" title="IMG-4116"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6aebf4112a2dafc7ad73b990200da9139a74fcb6_2_666x500.jpeg" alt="IMG-4116" data-base62-sha1="ffS8ESHh1HN8u0nT3eMABEAbnpA" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6aebf4112a2dafc7ad73b990200da9139a74fcb6_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6aebf4112a2dafc7ad73b990200da9139a74fcb6_2_999x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/a/6aebf4112a2dafc7ad73b990200da9139a74fcb6_2_1332x1000.jpeg 2x" data-dominant-color="33332F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG-4116</span><span class="informations">1920×1440 245 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> (No load)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/0/8080f946c538b5f1440f84da94adad99214a696c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/8080f946c538b5f1440f84da94adad99214a696c" title="IMG-4117"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/8080f946c538b5f1440f84da94adad99214a696c_2_666x500.jpeg" alt="IMG-4117" data-base62-sha1="ikNrlROlpcGa7SsS1qKSzSdt1XS" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/8080f946c538b5f1440f84da94adad99214a696c_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/8080f946c538b5f1440f84da94adad99214a696c_2_999x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/8080f946c538b5f1440f84da94adad99214a696c_2_1332x1000.jpeg 2x" data-dominant-color="353431"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG-4117</span><span class="informations">1920×1440 265 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> (single model training)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/6/d6b9872622659644d5a37ae7caa41aae30ebd24f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d6b9872622659644d5a37ae7caa41aae30ebd24f" title="IMG-4118"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6b9872622659644d5a37ae7caa41aae30ebd24f_2_666x500.jpeg" alt="IMG-4118" data-base62-sha1="uDxFz6HJhphwyaAsXTXbrfoE1rx" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6b9872622659644d5a37ae7caa41aae30ebd24f_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6b9872622659644d5a37ae7caa41aae30ebd24f_2_999x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/6/d6b9872622659644d5a37ae7caa41aae30ebd24f_2_1332x1000.jpeg 2x" data-dominant-color="31302D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG-4118</span><span class="informations">1920×1440 252 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div> (two independent models training)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/0/a03ff441d8c67600fbeff65163f160db214afc64.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/a03ff441d8c67600fbeff65163f160db214afc64" title="IMG-4119"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a03ff441d8c67600fbeff65163f160db214afc64_2_666x500.jpeg" alt="IMG-4119" data-base62-sha1="mRDpOuVUgVaSwMBmndQ6eirWso4" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a03ff441d8c67600fbeff65163f160db214afc64_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a03ff441d8c67600fbeff65163f160db214afc64_2_999x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/0/a03ff441d8c67600fbeff65163f160db214afc64_2_1332x1000.jpeg 2x" data-dominant-color="33322E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG-4119</span><span class="informations">1920×1440 240 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>  (two independent models training, another screenshot after few seconds)</p>

jaiabhayk: <p><a class="mention" href="/u/j_johnson">@J_Johnson</a> , <a class="mention" href="/u/ptrblck">@ptrblck</a>  , Please let me know if you have any suggestions ?</p>

J_Johnson: <p>99% utilization.</p>
<p>It means that your GPU compute cores are running at max speed. Memory is only one bottleneck. Calculation speed is the other. You’ve maxed out the latter. That’s why you’re getting a slow down on both models.</p>

jaiabhayk: <p><a class="mention" href="/u/j_johnson">@J_Johnson</a> , My bad (i did not realize that). I verified that with smaller network, it’s not an issue and works as expected. Thanks for your help !</p>

ptrblck: <p>The GPU util. in <code>nvidia-smi</code> reports the relative time where any kernel was using the GPU over the sample period. A high util. shows that kernels are packed, but does not give you any information about the occupancy of the compute resources. You should thus profile your code, check the needed resources (sm count, registers etc.) of your kernels and make sure they can overlap.</p>