Failure of Tesla K80 for training

koklimabc: <p>I’d ever experienced from Tesla K80 and gave comment that it is not suitable for data training</p>
<p>First things it could ended unknown error and second halt at the end of epochs once using pytorch lightling Trainer. I’d no issues on normal GTX970 card and faster as compare to it.</p>
<p>Anyone could fix it?</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://developer.nvidia.com/blog/increase-performance-gpu-boost-k80-autoboost/">
  <header class="source">
      <img src="https://developer-blogs.nvidia.com/wp-content/themes/nvidia/dist/images/favicon_300a1064.ico" class="site-icon" width="48" height="48">

      <a href="https://developer.nvidia.com/blog/increase-performance-gpu-boost-k80-autoboost/" target="_blank" rel="noopener nofollow ugc" title="02:19PM - 17 November 2014">NVIDIA Technical Blog – 17 Nov 14</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:179/115;"><img src="https://developer-blogs.nvidia.com/wp-content/uploads/2014/11/GPUBoost_thumb.png" class="thumbnail" width="179" height="115"></div>

<h3><a href="https://developer.nvidia.com/blog/increase-performance-gpu-boost-k80-autoboost/" target="_blank" rel="noopener nofollow ugc">Increase Performance with GPU Boost and K80 Autoboost | NVIDIA Technical Blog</a></h3>

  <p>NVIDIA® GPU Boost™ is a feature available on NVIDIA® GeForce® and Tesla® GPUs that boosts application performance by increasing GPU core and memory clock rates when sufficient power and thermal…</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>How to boost it up?</p>

J_Johnson: <p>Are you using proper cooling?</p>

ptrblck: <aside class="quote no-group" data-username="koklimabc" data-post="1" data-topic="193344">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/a183cd/48.png" class="avatar"> koklimabc:</div>
<blockquote>
<p>and gave comment that it is not suitable for data training</p>
</blockquote>
</aside>
<p>This sounds misleading as K80s were heavily used in e.g. Colab for all kinds of ML/DL use cases.</p>
<aside class="quote no-group" data-username="koklimabc" data-post="1" data-topic="193344">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/a183cd/48.png" class="avatar"> koklimabc:</div>
<blockquote>
<p>Anyone could fix it?</p>
</blockquote>
</aside>
<p>Unsure what exactly is failing as mentioning random errors is not really actionable.<br>
However, note that K80 support is dropped in the latest CUDA releases as Keppler GPUs are too old.</p>

koklimabc: <p>So not proper cooling will degrade the faster performance?<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/3/c30c9e45c45d9dd42fbb587c6b48052c0219f963.jpeg" alt="fan" data-base62-sha1="rPu8KQburE2CJMvrqcgOY44zBxF" width="418" height="422"><br>
FYI, i’m using this fan but change it from 12 volt to 5 volt for simply blow hot air to outside only (not cool air to inside).</p>

J_Johnson: <p>Are you using the K80 in a proper server or a desktop? You may get a lot of problems in desktop as it’s not designed for that.</p>
<p>You need something like a <a href="https://www.supermicro.com/products/archive/system/sys-2027gr-trf" class="inline-onebox" rel="noopener nofollow ugc">Supermicro | Products | SuperServers | 2U | 2027GR-TRF</a> which will have proper cooling, power, and connectivity. Can likely find on eBay. And get the proper RAM and CPUs. And then you will also need to setup IPMI to run the fans at the appropriate time.</p>

koklimabc: <aside class="quote no-group" data-username="J_Johnson" data-post="5" data-topic="193344">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_johnson/48/55494_2.png" class="avatar"> J_Johnson:</div>
<blockquote>
<p>IPMI</p>
</blockquote>
</aside>
<p>FYI， It is desktop used :), maybe i need modify to improve cooling performance.</p>

J_Johnson: <p>I think we’ve found you’re problem. You’ll get quite mixed results on using a K80 in a desktop. I speak from experience.</p>
<p>At least in a proper server, you can go days without an error interrupting training. But just note server fans are about as noisy as a prop plane.</p>

koklimabc: <p>Yea, agree, I’d recently tested out 12 volt for my fan (before that I swap to 5 volt under molex connector), then all trainings are runnning smooth to the end without any problem only a lot of noise coming from these two fans:).</p>
<p>Cooling is absolute important for computing device in order to train it without any problems.</p>