Slow model training on my local RTX 4070 comp. to Google Colab Tesla T4

Aeon_Lightyear: <p>Hello everyone. I’m a complete noob and just getting my feet wet with Pytorch.</p>
<p>I’m using Anaconda, version: 23.7.4<br>
Cuda version: 12.2.140<br>
Python version : 3.11.5<br>
Pytorch version: 2.1</p>
<p>I was following some tutorials, and have been using a free Google Colab account until now. Recently bought a new RTX 4070 and was very excited to stretch it’s legs. However, I was disappointed to see that it’s actually slower than a free Google Colab instance with a Tesla T4.</p>
<p>The model I used for testing is taken from one of the tutorials. Here it is:<br>
<a href="https://colab.research.google.com/drive/18JoMH3YIVPKboGECmxSwbmtTRUGeMKY1?usp=sharing" rel="noopener nofollow ugc">Google Drive Link to Model</a></p>
<p>It takes around 3 minutes to hit 100000 epochs on T4 and on my GPU it takes around 3.5 minutes.</p>
<p>Even though the task manager shows the GPU load at 92%, there are some weird things going on.</p>
<p>When I train the model locally on my GPU, CPU0 is getting loaded 100%, while other cores are not stressed much.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/d/5d23a16e53dc4317ef08a25c5aab162e538583af.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5d23a16e53dc4317ef08a25c5aab162e538583af" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d23a16e53dc4317ef08a25c5aab162e538583af_2_325x500.jpeg" alt="image" data-base62-sha1="dhWK3QXRxVBQa8kh03baU2F1Zin" width="325" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d23a16e53dc4317ef08a25c5aab162e538583af_2_325x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d23a16e53dc4317ef08a25c5aab162e538583af_2_487x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/d/5d23a16e53dc4317ef08a25c5aab162e538583af_2_650x1000.jpeg 2x" data-dominant-color="313435"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">792×1218 72.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Although it is loading a single core, torch.get_num_threads() returns 8. I tried setting it to torch.set_num_threads(os.cpu_count()) (which is 16), it did not change anything. Tried setting it to 1 also, the same result.</p>
<p>GPU temperature, GPU clock and memory clock are all extremely low for 90+% load, as you can see on the picture above.</p>
<p>It’s hovering around 500-700Mhz, when it is capable of 3Ghz.</p>
<p>Theoretically RTX 4070 should be much faster right? More cuda cores, higher frequencies etc. Yes T4 has more VRAM but, I’m hardly using 1Gig of VRAM anyways. I think I’m CPU limited, but even if that’s the case, I suspect that the single core the free Google Colab account gives me doesn’t have better single core performance than my overclocked Ryzen 5700. On the other hand GPU frequency behavior is super weird too.</p>
<p>What might be my problem? Thank you for your help.</p>

ptrblck: <p>Yes, you might be CPU-limited or the bottleneck might come from another part of the code, e.g. data loading. Create a full visual profile, e.g. via Nsight Systems, and compare both profiles to see where the bottleneck is. Alternatively, you could also narrow down the issue further by profiling separate parts of the code in isolation.</p>

Aeon_Lightyear: <p>I have additional information.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/7/57c5f7a0902457fdc02daad7acec9925e835d4a0.png" data-download-href="https://discuss.pytorch.org/uploads/default/57c5f7a0902457fdc02daad7acec9925e835d4a0" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/7/57c5f7a0902457fdc02daad7acec9925e835d4a0.png" alt="image" data-base62-sha1="cwtGl48PVsooc0uSge6B0awL2IE" width="666" height="499" data-dominant-color="232323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">811×608 17.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>nvidia-smi shows 0% utilization <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"> How the task manager measures the % load is a mystery.</p>
<p>Just as a sanity check, I set the device to CPU and run 100000 epochs couple times. Lo and behold, CPU is much faster <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=12" title=":smiley:" class="emoji" alt=":smiley:" loading="lazy" width="20" height="20"> , solved the same model in 1 minute 40-45 seconds consistently every run. Also all cores were being utilized.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5cabc7827161490a233720c18cff74a3f0a2241d.png" data-download-href="https://discuss.pytorch.org/uploads/default/5cabc7827161490a233720c18cff74a3f0a2241d" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5cabc7827161490a233720c18cff74a3f0a2241d.png" alt="image" data-base62-sha1="ddNXH1XbrxFlUIp2ZgFdEu42gAR" width="519" height="500" data-dominant-color="36535E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">796×766 24.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So I guess there are models that are better trained on CPU’s for some reason?</p>
<p>I have no idea how to use Nsight, just poked around on it’s UI. I gave the vs code executable with command line argument -r and pointed the working directory to where the .ipynb is. Launched it, ran the notebook, said no kernel was profiled. I guess it doesn’t work like that. I think it works with the vs code add-on and I should  run the code in some sort of a debugging mode with the help of the extension. I need to read some documentation, this tool is alien to me as of now.</p>

barista: <p><a class="mention" href="/u/aeon_lightyear">@Aeon_Lightyear</a>, did you make any progress?<br>
What OS are you using?<br>
My old 2017 AMD Ryzen Threadripper 2950X is also showing its age, and my RTX 4090 does not live up to my expectations.</p>

ptrblck: <p>The same recommendations is still valid. Profile your code properly as you are speculating otherwise as seen in previous answers.</p>