Torch crashing nVidia driver with this simple example :-( Help

burhop: <p>My Windows PC is crashing doing some Llama2 training and I’m really struggling to find a solution.  This training example will reproduce it… It starts fine but then crashes the PC about an hour in during the train</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.datacamp.com/tutorial/fine-tuning-llama-2">
  <header class="source">
      <img src="https://www.datacamp.com/marketing-backgrounds/favicons/favicon-196x196.png" class="site-icon" width="196" height="196">

      <a href="https://www.datacamp.com/tutorial/fine-tuning-llama-2" target="_blank" rel="noopener nofollow ugc">datacamp.com</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:689/459;"><img src="https://images.datacamp.com/image/upload/v1707148423/datarhys_an_absurdist_oil_painting_of_a_robot_llama_being_repai_f6776542_9528_41fd_ad81_66b13f5a7ddf_68cd2a6b56.png" class="thumbnail" width="689" height="459"></div>

<h3><a href="https://www.datacamp.com/tutorial/fine-tuning-llama-2" target="_blank" rel="noopener nofollow ugc">Fine-Tuning LLaMA 2: A Step-by-Step Guide to Customizing the Large Language...</a></h3>

  <p>Learn how to fine-tune Llama-2 using new techniques to overcome memory and computing limitations to make open-source large language models more accessible</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Windows 10 32Gig<br>
RTX 4080 32Gig<br>
Python 3.11.8<br>
torch 2.2.1+cu121<br>
torchaudio 2.2.1+cu121<br>
torchvision 0.17.1</p>
<p>NVidia Drivers 551.61 (release date: 02/22/2024)</p>
<p>The crash dump files all mention nvlddmkm.sys. I have the latest driver but have tried some previous versions too.  I’ve also tried reducing the batch size to 2 and to 1. I’ve posted on the nVidia board since I’m crashing the PC but no luck here</p>
<aside class="onebox discoursetopic" data-onebox-src="https://forums.developer.nvidia.com/t/best-nvidia-driver-for-pytorch-im-crashing/286221">
  <header class="source">
      <img src="https://global.discourse-cdn.com/nvidia/optimized/2X/0/0372ccc95874f71d7fbff64bbbff6f8c69dd850b_2_32x32.png" class="site-icon" width="32" height="32">

      <a href="https://forums.developer.nvidia.com/t/best-nvidia-driver-for-pytorch-im-crashing/286221" target="_blank" rel="noopener nofollow ugc" title="01:34PM - 15 March 2024">NVIDIA Developer Forums – 15 Mar 24</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:150/80;"><img src="https://global.discourse-cdn.com/nvidia/original/3X/e/4/e41d27491ee72562cf68c340597ca95f587879bd.png" class="thumbnail" width="150" height="80"></div>

<div class="title-wrapper">
  <h3><a href="https://forums.developer.nvidia.com/t/best-nvidia-driver-for-pytorch-im-crashing/286221" target="_blank" rel="noopener nofollow ugc">Best Nvidia Driver for PyTorch? I'm crashing :-(</a></h3>
  <div class="topic-category">
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #76B900;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">AI &amp;amp; Data Science</span>
        </span>
      </span>
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #76B900;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">AI Foundation Models and Endpoints</span>
        </span>
      </span>
    <div class="topic-header-extra">
      <div class="list-tags">
        <div class="discourse-tags">
          <svg class="fa d-icon d-icon-tag svg-icon svg-string"><use href="#tag"></use></svg>
            <span class="discourse-tag simple">pytorch</span>
            <span class="discourse-tag simple">drivers</span>
        </div>
      </div>
    </div>
  </div>
</div>

  <p>Training my models keeps crashing my PC because of nvlddmkm.sys. My next thought is to experiment with other versions of the drivers (anyone got a link to where I can get them?)  A few more details in case there is any nVidia expert out there. ...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I’m not a pyTorch expert so if you have suggestions on parameters or other modules to test, that would be a big help.  In the mean time, I’ll give it a try on one of my LInux machines.</p>

ptrblck: <p>I doubt the issue is related to PyTorch, so you might want to follow up in the cross-post.</p>

burhop: <p>I hope you are right. Even if I am trying to use too much memory, the OS and drivers should kick back and error, not crash.</p>
<p>I ran the code on my Win11 32gig  with a RTX 2070 24gig machine and got the same crash. So at least I have eliminated hardware.   If I run the example code as is, it gives a memory error.   Changing the batch size from 4 to 1 let me get further and crashes both PCs.</p>
<p>A Linux machine with a 3080 is next.</p>

OracPrime: <p>I’m also seeing not a crash with a dump, but an instant Windows reboot when training a model with torch 2.2.2+cu121 and cuda 12.4.  Do I need to downgrade to cuda 12.1 or upgrade torch somehow?</p>

burhop: <p>I did upgrade to the NVIDIA drivers that just came out but no joy.</p>
<p>Setting PyTorch to “cpu” allows it to work (if you have lots of time)</p>
<p>Running it on the Linux machine didn’t get far as I immediately get an error message saying it can’t access enough GPU memory.</p>

OracPrime: <p>Thanks.   I don’t think I’ve got the patience for cpu mode.  I’ll keep playing and let you know if I find a fix.  Both upgrading PyTorch to 12.4 (by building locally from source) and persuading CUDA to downgrade look painful…</p>