Unable to Utilize GPUs with PyTorch on p2.xlarge EC2 Instances

Harsh_Mittal: <p>Hi there!</p>
<p>I’m struggling to utlize the GPUs from p2.xlarge EC2 instance. I’ve installed nvidia driver &amp; dkms as well.</p>
<p>Check out the snapshot where I’m getting a warning from Pytorch &amp; the GPU system overview got from running ‘nvidia-smi’ command.</p>
<p>Here are some details.</p>
<p>EC2 Instance: p2.xlarge<br>
AMI: Ubuntu<br>
Torch Version: 2.X<br>
Nvidia driver: 470.xxx<br>
Cuda Version: 11.4</p>
<p>Looking forward to the solution <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
Thanks!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bffd1dc1c0b9499e93d997ec736289c165e9cdf3.png" data-download-href="https://discuss.pytorch.org/uploads/default/bffd1dc1c0b9499e93d997ec736289c165e9cdf3" title="Screenshot 2024-04-25 at 12.55.18 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bffd1dc1c0b9499e93d997ec736289c165e9cdf3_2_690x264.png" alt="Screenshot 2024-04-25 at 12.55.18 PM" data-base62-sha1="ropuKdpCwEaiBIMffpZTYdNC0Sv" width="690" height="264" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bffd1dc1c0b9499e93d997ec736289c165e9cdf3_2_690x264.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bffd1dc1c0b9499e93d997ec736289c165e9cdf3_2_1035x396.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/f/bffd1dc1c0b9499e93d997ec736289c165e9cdf3_2_1380x528.png 2x" data-dominant-color="121313"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-04-25 at 12.55.18 PM</span><span class="informations">2880×1102 276 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

marksaroufim: <p>So the problem is your NVIDIA driver is too old so you either need to upgrade your CUDA driver or downgrade your pytorch version. For the former you can follow guides like this <a href="https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html" class="inline-onebox">CUDA Installation Guide for Linux</a></p>
<p>Or if you want to downgrade your torch version you can see what’s supported here <a href="https://download.pytorch.org/whl/torch/">https://download.pytorch.org/whl/torch/</a></p>
<p>And when you found the version you like can you pip install the wheel directly</p>

Harsh_Mittal: <p>Hi <a class="mention" href="/u/marksaroufim">@marksaroufim</a>,</p>
<p>After searching I got that Tesla K80 GPUs are being supported by CUDA toolkit 10.0 version &amp; 10.0 CUDA toolkit is only available for Ubuntu 18, 16 &amp; 14.</p>
<p>The available Ubuntu options that I have are 20, 22 &amp; 24.</p>
<p>Seems like K80s are very old &amp; old architectures (Ubuntu, CUDA toolkit) supports them.</p>
<p>Do you have any suggestions for me around how can I use K80? Or do I need to go for the some what new GPUs?</p>
<p>Also I’m thinking about degrading the Pytorch that’s compatible with the current CUDA version I’ve (11.4) - can you help me with finding the Pytorch version that will work with CUDA 11.4?</p>
<p>Would appreciate your help.<br>
Thanks</p>

ptrblck: <aside class="quote no-group" data-username="Harsh_Mittal" data-post="3" data-topic="201584">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/harsh_mittal/48/68825_2.png" class="avatar"> Harsh_Mittal:</div>
<blockquote>
<p>Tesla K80 GPUs are being supported by CUDA toolkit 10.0 version</p>
</blockquote>
</aside>
<p>Newer CUDA toolkits, up to 11.8, also support Keppler architectures.</p>
<aside class="quote no-group" data-username="Harsh_Mittal" data-post="3" data-topic="201584">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/harsh_mittal/48/68825_2.png" class="avatar"> Harsh_Mittal:</div>
<blockquote>
<p>can you help me with finding the Pytorch version that will work with CUDA 11.4?</p>
</blockquote>
</aside>
<p>PyTorch binaries ship with their own CUDA runtime dependencies and your locally installed CUDA toolkit won’t be used unless you build PyTorch from source or a custom CUDA extension.</p>

Harsh_Mittal: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thank you for the response.</p>
<p>So I don’t have to install CUDA toolkit because Pytorch has it’s own CUDA implementation (correct me If I’m wrong).</p>
<p>You can see the snapshot <a class="mention" href="/u/ptrblck">@ptrblck</a> - what actions do you suggest to solve this issue of ‘Old Nvidia Driver’.</p>
<p>I just want to setup the P2 instance in a way that the latest pytorch can utilise the GPUs - I just want to make it happen <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Looking forward to your response.</p>
<p>Thank You</p>

ptrblck: <p>If your setup has issues communicating with the GPU, you might indeed want to update the driver although I would assume the PyTorch + CUDA 11.8 binaries should just work with the older driver. Are you able to run any other CUDA application in this setup? Just to make sure it’s indeed working.</p>