RuntimeError: CUDA error: unspecified launch failure // Neural Amp Modeler

giacomopas: <p>Hi everyone,</p>
<p>First of all, I want to apologize as I’m not a programmer or developer, so I may not have a complete understanding at the same level as many of you here.</p>
<p>I’m using Anaconda + PyTorch to create guitar profiles through Neural Amp Modeler, which performs local training on this infrastructure.</p>
<p>I recently switched from an Intel 14900K to an AMD 9950X, and now I can’t complete the training, something I’ve never had issues with before.<br>
GPU remained the same: RTX4070.</p>
<p>I’ve already tried uninstalling Anaconda completely, reinstalling it, and reinstalling PyTorch, but no luck.</p>
<p>Does anyone have any ideas on how to solve this? I really need to get back to work on this project.</p>
<p>I’m linking two text files with the content of the Anaconda Prompt here: <a href="https://www.dropbox.com/scl/fo/4cjvob5tchjq23ls2tfy1/AJ0GtvlA2wVv9mCMq5L0nJY?rlkey=pg6tkeoolog6fodl7qm7qjajf&amp;dl=0" class="inline-onebox" rel="noopener nofollow ugc">Dropbox</a></p>
<p>Thanks so much for your help!</p>

giacomopas: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> some kind of help might be highly appreciated! Thanks so much!</p>

giacomopas: <p>I also did the verification you have here: <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/8/680cc69ee1fbc285720f7cd0c73d9095b53c3fcf.png" data-download-href="https://discuss.pytorch.org/uploads/default/680cc69ee1fbc285720f7cd0c73d9095b53c3fcf" title="WindowsTerminal_Pt04LgefM3"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/8/680cc69ee1fbc285720f7cd0c73d9095b53c3fcf.png" alt="WindowsTerminal_Pt04LgefM3" data-base62-sha1="eQt1NNEfNJK5Lz6fiWLgLGYNAPJ" width="690" height="388" data-dominant-color="121212"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">WindowsTerminal_Pt04LgefM3</span><span class="informations">1113×626 13.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks to anyone who can help!</p>

ptrblck: <p>Could you check if <code>dmesg</code> is reporting any <code>Xid</code> error codes?<br>
Also, are you able to move the tensor from your example to the GPU as the verification is incomplete?</p>

giacomopas: <p>Thanks for your answer!<br>
As I wrote, I’m really unexperienced about Anaconda/Pytorch.<br>
Is there a guide to check what you asked me?</p>
<p>Thanks so much…</p>

ptrblck: <p>In your code, allocate a tensor on the GPU via:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(16, 16, device="cuda")
print(x)
print(torch.matmul(x, x))
</code></pre>
<p>and make sure the output shows a valid tensor without raising any errors.</p>
<p>If so, run <code>dmesg -T | grep -i xid</code> in your terminal next and check if any errors are raised and correlate to failures in your script when checking the timestamp.</p>

giacomopas: <p>Just a quick update that solved the issue, just in case it might be useful for someone else.<br>
I was running XMP profile on my computer.</p>
<p>Disabling it fixed the issue without any other step.</p>

ptrblck: <p>Thanks for sharing your solution! How did you figure out the issues were caused by XMP?</p>

giacomopas: <p>I couldn’t understand why the system was exactly the same, but I was having issues. So, after some research, I noticed instabilities with XMP1 on the AMD platform and decided to do an A/B comparison of the training process with and without it. With it, I had the issue I mentioned earlier, but without it, I was able to complete the training without problems!</p>