Failed CUDA graph capture leaves default stream in invalid state

username: RaulPPelaez
cooked: <p>The following MRE results in an unexpected error:</p>
<pre><code class="lang-python">    stream = torch.cuda.Stream(-1)
    torch.cuda.synchronize()
    try:
        graph = torch.cuda.CUDAGraph()
        with torch.cuda.graph(graph, stream=stream):
            torch.cuda.synchronize()
    except RuntimeError as e:
        print(e)
    torch.cuda.synchronize()
    del stream
    x = torch.randn(10, device="cuda")

</code></pre>
<p>Naturally, the exception printed is related to synchronize not being something that can be used inside a CUDA graph:</p>
<pre><code class="lang-auto">CUDA error: operation failed due to a previous error during capture
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
</code></pre>
<p>Unexpectedly for me is that the randn line also fails:</p>
<pre><code class="lang-python">&gt;       x = torch.randn(10, device="cuda")
E       RuntimeError: CUDA generator expects graph capture to be underway, but the current stream is not capturing.
</code></pre>
<p>AFAIK this happens because of this line here:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/af1d437654aa7fd5aa4a5d30ae2f2e9af34c7765/aten/src/ATen/cuda/CUDAGeneratorImpl.cpp#L288">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/af1d437654aa7fd5aa4a5d30ae2f2e9af34c7765/aten/src/ATen/cuda/CUDAGeneratorImpl.cpp#L288" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/af1d437654aa7fd5aa4a5d30ae2f2e9af34c7765/aten/src/ATen/cuda/CUDAGeneratorImpl.cpp#L288" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/af1d437654aa7fd5aa4a5d30ae2f2e9af34c7765/aten/src/ATen/cuda/CUDAGeneratorImpl.cpp#L288</a></h4>



    <pre class="onebox"><code class="lang-cpp">
      <ol class="start lines" start="278" style="counter-reset: li-counter 277 ;">
          <li>* each thread. It is the user's responsibility to make sure the increment</li>
          <li>* for philox is never smaller than the number of curand() calls. Increment</li>
          <li>* value &gt; the number of curand() calls won't harm but anything less would mean</li>
          <li>* that you would be reusing random values from previous calls.</li>
          <li>*</li>
          <li>* See Note [Acquire lock when using random generators]</li>
          <li>*/</li>
          <li>PhiloxCudaState CUDAGeneratorImpl::philox_cuda_state(uint64_t increment) {</li>
          <li> // rounds increment up to the nearest multiple of 4</li>
          <li> increment = ((increment + 3) / 4) * 4;</li>
          <li class="selected"> if (at::cuda::currentStreamCaptureStatus() != at::cuda::CaptureStatus::None) {</li>
          <li>   TORCH_CHECK(graph_expects_this_gen_,</li>
          <li>               "philox_cuda_state for an unexpected CUDA generator used during capture. "</li>
          <li>               CAPTURE_DEFAULT_GENS_MSG);</li>
          <li>   // see Note [Why enforce RNG offset % 4 == 0?]</li>
          <li>   TORCH_INTERNAL_ASSERT(this-&gt;offset_intragraph_ % 4 == 0);</li>
          <li>   uint32_t offset = this-&gt;offset_intragraph_;</li>
          <li>   TORCH_INTERNAL_ASSERT(this-&gt;offset_intragraph_ &lt;=</li>
          <li>                         std::numeric_limits&lt;uint32_t&gt;::max() - increment);</li>
          <li>   this-&gt;offset_intragraph_ += increment;</li>
          <li>   return PhiloxCudaState(this-&gt;seed_extragraph_,</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>When capture fails the stream is left with the “Invalidated” status:<br>
<a href="https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1g0ec4aacc66fe76c145252d12b189e233" class="onebox" target="_blank" rel="noopener nofollow ugc">https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1g0ec4aacc66fe76c145252d12b189e233</a></p>
<p>A failed capture should not invalidate the CUDA context. I wonder why the stream used during capture is also being used in torch.randn.<br>
Am I doing something wrong here?<br>
If so, how can I recover from a failed capture?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="RaulPPelaez" data-post="1" data-topic="180946">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/raulppelaez/48/56443_2.png" class="avatar"> RaulPPelaez:</div>
<blockquote>
<p>A failed capture should not invalidate the CUDA context.</p>
</blockquote>
</aside>
<p>I don’t think this is the case here as it seems an internal <code>Runtime</code> error is raised assuming that the capture might still be ongoing due to it failed status.<br>
What’s your use case assuming a capture fails as I don’t think it’s easy to recover from internal asserts?</p>

username: RaulPPelaez
cooked: <p>Pytorch not supporting recovering from an unsuccessful capture is good enough for me. I just wanted to make sure this was by design.<br>
Regarding my use case, I was thinking on having the possibility of falling back to eager mode if the capture of a module was unsuccessful.<br>
Thanks.</p>

username: ptrblck
cooked: <p>OK, I see. Unfortunately, the error looks like a valid CUDA assert and might thus corrupt the CUDA context so cannot be recovered.</p>