akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Bug: scaled_dot_product_attention slower than matmul

JosAr: <p>Hi,<br>
I have written this before and believe I found a / the same bug again: I have tried replacing my manual implementation (or the one given here <a href="https://docs.pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional.scaled_dot_product_attention — PyTorch 2.8 documentation</a> as the standard version) with the efficient scaled_dot_product_attention and received much slower runtimes.</p>
<p>This time I have tried it with tensors of the shape (batch size, seq_length, dim): [28800, 8, 48], [1, 28800, 8, 48] or [28800, 1, 8, 48] (all the same, but an empty dimension added to hopefully suit the description of SDPA). However None of them were faster, but slower by factor 5 on my A100. Note that the added empty dimension is there to simulate different heads, as heads should be treated exactly as batches; however the results were devastating.</p>
<p>Then I tried it with another shape that occurs often in my project:</p>
<p>[115200, 8, 24] (like this SDPA was 5 times slower)</p>
<p>[115200, 1, 8, 24], and</p>
<p>[1, 115200, 8, 24]. However the two last attempts gave me this:</p>
<p><code>torch.AcceleratorError: CUDA error: invalid configuration argument</code><br>
<code>Search for cudaErrorInvalidConfiguration' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information. CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with TORCH_USE_CUDA_DSA` to enable device-side assertions.</code></p>
Computation error due to different data types

Rewant_Angira: <p>I am currently building a ANN model using pytorch  on a diabetes dataset so first i converted every feature into float64 and then did the front prop and back prop but during the training loop i am getting an error  mat1 and mat2 must have the same dtype, but got Double and Float.</p>
<p>so how can i fix it and whats the problem here</p>

valerian.rey: <p>Hey! It’s probably because your features are stored in Double (torch.float64) tensors, and your model parameters are stored on the default Float (torch.float32) tensors. Both must be of the same precision for the forward pass to work.</p>
<p>You can do:</p>
<pre data-code-wrap="python"><code class="lang-python">model = model.to(dtype=torch.float64)
</code></pre>
<p>to use float64 precision for your model (this will use twice as much memory though), or you can use:</p>
<pre data-code-wrap="python"><code class="lang-python">input = input.to(dtype=torch.float32)
</code></pre>
<p>to use float32 precision for your inputs (or simply create them as float32 directly rather than float64).</p>

GerryMan: <p>Did you maybe only change the data to float64 and not the model itself??</p>
Fft2/rfft2 makes CPU memory keep increasing

xwghua: <p>I was trying to calculate B=torch.fft.rfft2(A) in a for loop on CPU. Although the speed is pretty fast compared to numpy.fft.rfft2, the CPU memory keeps increasing. This issue only happens with Pytorch 2.8.0, while the same code works perfectly with Pytorch 2.7.1. Anyone could by any chance know where the problem is? Many thanks!</p>

ptrblck: <p>I cannot reproduce any increase in memory using:</p>
<pre><code class="lang-auto">import torch

t = torch.rand(10, 10)
for _ in range(10000000000):
    rfft2 = torch.fft.rfft2(t)
</code></pre>