"RuntimeError: CUDA error: an illegal memory access was encountered" at loss.backward()

n-zalok: <p>Strated getting this error lately even on code that was executed successfully before.</p>
<p>My setup is GTX970 + Driver Version: 550.54.14  CUDA Version: 12.4 + Pytorch 2.6.0+cu124</p>
<pre><code class="lang-auto">for epoch in range(1, self.epochs + 1):
    self.model.train()
    loop = tqdm(self.train_loader, desc=f"Epoch {epoch}/{self.epochs}", leave=False)

        for step, batch in enumerate(loop):
            # Move everything to device
            batch = {k: v.to(self.device) for k, v in batch.items()}

            # Forward
            outputs = self.model(
                    input_ids=batch["input_ids"],
                    attention_mask=batch["attention_mask"]
            )
            logits = outputs.logits
            loss = self.criterion(logits, batch["labels"])
            loss = loss / self.accumulation_steps

            # Backward
            nvtx.range_push("loss_backward")
            loss.backward()
            nvtx.range_pop()
</code></pre>
<p>The error is inconsistent (shows up at random batches) but always shows up at the same line (loss.backward())</p>
<pre><code class="lang-auto">  File "/mnt/ssd/ME/NLP/Math/Math-Explainer/DistilBERT.py", line 114, in train
    loss.backward()
    ~~~~~~~~~~~~~^^
  File "/home/noor-zalouk/.local/lib/python3.13/site-packages/torch/_tensor.py", line 626, in backward
    torch.autograd.backward(
    ~~~~~~~~~~~~~~~~~~~~~~~^
        self, gradient, retain_graph, create_graph, inputs=inputs
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/noor-zalouk/.local/lib/python3.13/site-packages/torch/autograd/__init__.py", line 347, in backward
    _engine_run_backward(
    ~~~~~~~~~~~~~~~~~~~~^
        tensors,
        ^^^^^^^^
    ...&lt;5 lines&gt;...
        accumulate_grad=True,
        ^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/noor-zalouk/.local/lib/python3.13/site-packages/torch/autograd/graph.py", line 823, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        t_outputs, *args, **kwargs
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
    )  # Calls into the C++ engine to run the backward pass
    ^
RuntimeError: CUDA error: an illegal memory access was encountered
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

</code></pre>
<p>What I tried:<br>
1- creating a new environment<br>
2- using a different Pytorch version<br>
3- using a different Nvidia driver<br>
4- checking gpu’s vram using cuda_memtest (ran for several hours with no problem)<br>
5- reinstalling the OS (Debian 12)<br>
6- lowering -Xptxas level as mentioned <a href="https://forums.developer.nvidia.com/t/illegal-memory-access-but-memcheck-and-sanitizer-return-0-error/169851/2" rel="noopener nofollow ugc">here</a> (only makes the code run longer)</p>
<p>Runing with compute-sanitizer succeeded without any errors</p>

ptrblck: <aside class="quote no-group" data-username="n-zalok" data-post="1" data-topic="223383">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/n-zalok/48/78304_2.png" class="avatar"> n-zalok:</div>
<blockquote>
<p>Runing with compute-sanitizer succeeded without any errors</p>
</blockquote>
</aside>
<p>That’s unexpected. In any case, since your Maxwell GPU is quite old by now, did you try to install PyTorch binaries built with CUDA 11.8?</p>