First call to eigh(), but not subsequent calls, gives "cusolver error: CUSOLVER STATUS EXECUTION FAILED"

KFrank: <p>Hi Forum!</p>
<p>I see a cuda error (in a particular situation) the first time I call <code>eigh()</code>,<br>
but not on subsequent calls.  (This is on the current stable, 1.11.0, but<br>
I also see it on the latest nightly, 1.13.0.dev20220626.)</p>
<p>By “first time” I mean that I start a new python session and then run the<br>
script below – the first call fails and the second succeeds.  If I rerun the<br>
script in the same python session, both calls succeed.</p>
<p>Here is the code that reliably reproduces the issue for me:</p>
<pre><code class="lang-python">import torch

print (torch.__version__)
print (torch.version.cuda)
print (torch.cuda.get_device_name())

_ = torch.manual_seed (2022)

matrices = torch.randn (10000, 200, 200, device = 'cuda')

print ('eigh first attempt ...')
try:
    torch.linalg.eigh (matrices)
except Exception as e:
    print ('exception caught:', str (e))

print ('eigh second attempt ...')
try:
    torch.linalg.eigh (matrices)
except Exception as e:
    print ('exception caught:', str (e))
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-nohighlight">1.11.0
11.3
GeForce GTX 1050 Ti
eigh first attempt ...
exception caught: cusolver error: CUSOLVER_STATUS_EXECUTION_FAILED, when calling `cusolverDnSsyevj( handle, jobz, uplo, n, A, lda, W, work, lwork, info, params)`. This error may appear if the input matrix contains NaN.
eigh second attempt ...
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

ptrblck: <p>Hi <a class="mention" href="/u/kfrank">@KFrank</a>,</p>
<p>thanks for creating the issue and code snippet!<br>
So far I was unable to reproduce the issue using <code>1.13.0.dev20220626+cu113</code> on a P100, which is of course not the same device you are using (same architecture though).<br>
While I’m trying to grab a 1050Ti (or a more comparable GPU) could you try to install the latest CUDA11.6 nightlies and see if you would still hit the issue?<br>
This should work:</p>
<pre><code class="lang-auto">pip install --pre torch --extra-index-url https://download.pytorch.org/whl/nightly/cu116
</code></pre>

KFrank: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
<p>Yes, I will give the CUDA11.6 nightly a try.</p>
<p>I have been using conda exclusively for pytorch (except for that old 0.3.0<br>
version I was playing with a couple of years back), so I may need some<br>
coaching on pip.  Should I run that pip command within a conda environment<br>
to avoid conflicts with other versions of pytorch?</p>
<p>Or to ask it another way, how should I manage using conda on the one hand<br>
and pip on the other?</p>
<p>Thanks.</p>
<p>K. Frank</p>

ptrblck: <p>Yes, creating a new conda env and using the posted <code>pip install</code> command should still work (this is how I’m testing the binaries usually). In any case, let me post the <code>conda install</code> command later as an update in case you want to use these binaries.</p>

KFrank: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
<p>Thanks.  Running your <code>pip install</code> command in a new conda environment<br>
worked.</p>
<p>Using the cu116 nightly caused the first-call “cusolver error” to go away.<br>
Here is the output of the script I posted above:</p>
<pre><code class="lang-nohighlight">1.13.0.dev20220626+cu116
11.6
GeForce GTX 1050 Ti
eigh first attempt ...
eigh second attempt ...
</code></pre>
<p>As an aside, the gpu slowness (relative to the cpu) persists with the cu116<br>
nightly.</p>
<p>(Also, the error still happened with the cu113 nightly after I tried rebooting<br>
my machine just in case the gpu was in some confused state, although, for<br>
me, the symptom that gets fixed by rebooting is the gpu not working at all<br>
with pytorch.)</p>
<p>Best.</p>
<p>K. Frank</p>