akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Plans to support fp8 nn.Conv2d?

sipie800: <p>Now torch has supported fp8 tensor, and fp8 nn.Linear.<br>
Any plans to support fp8 nn.Conv2d and other regular modules ?<br>
If not, could anybody provide some clue about where I should start to custom the nn.Conv2d for fp8 ? Should I start from cudnn c++ coding ? I’ve no idea at all, thanks.</p>
Libtorch mpi distribution?

Fitanium: <p>few days ago, I plan to study libtorch distributed training(MPI backend), and found an example dist-mnist <a href="https://github.com/pytorch/examples/blob/main/cpp/distributed/dist-mnist.cpp" rel="noopener nofollow ugc">examples/cpp/distributed/dist-mnist.cpp at main · pytorch/examples · GitHub</a>, in which <code>ProcessGroupMPI</code> is used for distribution.</p>
<p>When I compile the source code, compiler reported two error messages:</p>
<ul>
<li><code>undefined reference to `c10d::ProcessGroupMPI::abort()'</code></li>
<li><code>undefined reference to `c10d::ProcessGroupMPI::createProcessGroupMPI(std::vector&lt;int, std::allocator&lt;int&gt; &gt;)'</code></li>
</ul>
<p>seems that mpi-distribution codes are not compiled in pre-compiled libtorch package, and/or which library I should link?</p>
<p>Thanks.</p>

H-Huang: <p>PyTorch binaries do not include an MPI implementation and you will have to build from source. There are many different implementations you can choose from:</p>
<p><a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html#:~:text=nvidia%2Dsmi.-,MPI%20Backend,-The%20Message%20Passing" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/dist_tuto.html#:~:text=nvidia-smi.-,MPI%20Backend,-The%20Message%20Passing</a></p>
<p>When building you should ensure that this flag USE_MPI=1, <a href="https://github.com/pytorch/pytorch/blob/main/setup.py#L91-L92" class="inline-onebox" rel="noopener nofollow ugc">pytorch/setup.py at main · pytorch/pytorch · GitHub</a></p>

Fitanium: <p>Thank you. Based on your suggestions, I manually compiled the PyTorch source code and enabled the GLOO and MPI backends. Now I can complete the compilation and linking process without any error messages.Thanks again.</p>
Torch.compile() causes RuntimeError: element 0 of tensors does not require

Joeyoung: <p>Hi,</p>
<p>In the second torch.autograd.grad() for second derivative (d2y_dx2 in the below example) I got the RuntimeError:element 0 of tensors does not require grad and does not have a grad_fn.<br>
I don’t get what the error means.The error happens only when torch.compile() is applied to ScalarNet. Any explanation will be appreciated!</p>
<pre><code class="lang-auto">input_scalar = torch.randn(1, device=device, requires_grad=True)


# Define the module
class ScalarNet(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.seq = torch.nn.Sequential(
            torch.nn.Linear(1, 5), torch.nn.Tanh(), torch.nn.Linear(5, 1)
        )

    def forward(self, x):
        # Ensure input has feature dim
        if x.ndim == 1:
            x = x.unsqueeze(-1)
        return self.seq(x).squeeze(-1)  # Squeeze output back to scalar per batch item


net_scalar = ScalarNet().to(device)

# Compile the module
try:
    print("Compiling scalar model...")
    compiled_net_scalar = torch.compile(net_scalar)
    print("Compilation successful.")
except Exception as e:
    print(f"Scalar model compilation failed: {e}")
    compiled_net_scalar = net_scalar  # Fallback

# Compute derivatives using the compiled module
try:
    # First derivative
    y = compiled_net_scalar(input_scalar)
    # Ensure grad_outputs matches y shape (should be (1,) if input_scalar is (1,))
    grad_output_y = torch.ones_like(y)
    dy_dx = torch.autograd.grad(
        y, input_scalar, grad_outputs=grad_output_y, create_graph=True
    )[0]
    print("First derivative computed.")

    # Second derivative
    # Ensure grad_outputs matches dy_dx shape (should be (1,))
    grad_output_dydx = torch.ones_like(dy_dx)
    # Use retain_graph=False if this is the last grad call needing this graph segment
    d2y_dx2 = torch.autograd.grad(
        dy_dx, input_scalar, grad_outputs=grad_output_dydx, create_graph=False
    )[0]
    print("Second derivative computed successfully!")
    print(f"d2y/dx2 = {d2y_dx2.item()}")

except Exception as e:
    print(f"\n--- ERROR during second derivative test ---")
    print(e)
    import traceback

    traceback.print_exc()

</code></pre>

Joeyoung: <p>Never mind. I can solve the problem by adding fullgraph=True.</p>
Clip Grad Norm on GPU without sync

PhysicsGaunt: <p>( Not sure about the category, sorry <img src="https://discuss.pytorch.org/images/emoji/apple/expressionless.png?v=12" title=":expressionless:" class="emoji" alt=":expressionless:" loading="lazy" width="20" height="20"> )</p>
<p>I am running a training on CUDA and have noticed that the clip_grad_norm function seems to be quite slow, I believe this is due to the fact that is uses is_nonzero somewhere ( I can not find where ) which synchronizes with the CPU. Is there a way to clip the grad norms without synchronizing with the CPU to save time?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/7/27a4df55aa196200c251751b802b721d473b5ae3.png" data-download-href="https://discuss.pytorch.org/uploads/default/27a4df55aa196200c251751b802b721d473b5ae3" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27a4df55aa196200c251751b802b721d473b5ae3_2_690x61.png" alt="image" data-base62-sha1="5EHRfmYt88JX2I1VDcKRG9xYTwT" width="690" height="61" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27a4df55aa196200c251751b802b721d473b5ae3_2_690x61.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27a4df55aa196200c251751b802b721d473b5ae3_2_1035x91.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/7/27a4df55aa196200c251751b802b721d473b5ae3_2_1380x122.png 2x" data-dominant-color="BB94C6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2111×187 25.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Chinggam26: <p>Hey <a class="mention" href="/u/physicsgaunt">@PhysicsGaunt</a> , would you be able to share which tool was used to gather that flame graph?<br>
I understand you are more interested in why it took a long time for device synchronization compared to actual compute, but I would like to believe that is more of a limitation for all kernels and should be quite small if there was enough data for the GPU to munch through and the behavior you are seeing would be just a slight overhead at best.</p>
<p>It would be great if the driver code could be shared/explained, as in what kind of params were passed to <code>clip_grad_norm_()</code> to get above results.</p>
<p>It looks like this is the <code>is_nonzero()</code>: <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/prim_native_functions.cpp#L14" class="inline-onebox" rel="noopener nofollow ugc">pytorch/aten/src/ATen/native/prim_native_functions.cpp at main · pytorch/pytorch · GitHub</a> that gets invoked and then it calls <code>_local_scalar_dense()</code> which then waits for sync to happen here: <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/cuda/CUDAScalar.cu#L46" class="inline-onebox" rel="noopener nofollow ugc">pytorch/aten/src/ATen/native/cuda/CUDAScalar.cu at main · pytorch/pytorch · GitHub</a></p>
Error: accessing tensor output of CUDAGraphs that has been overwritten by a subsequent run

barakugav: <p>Im trying to compile my model using torch.compile with mode=‘reduce-overhead’ on a machine with CUDA, and no matter what I do I get a runtime error saying “Error: accessing tensor output of CUDAGraphs that has been overwritten by a subsequent run”.<br>
Here is a minimal example reproducing the issue:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn

device = 'cuda'
model = nn.Linear(10, 1).to(device)
model = torch.compile(model, fullgraph=True, mode='reduce-overhead')

for _ in range(5):
    torch.compiler.cudagraph_mark_step_begin()

    input = torch.randn((10,), dtype=torch.float32, requires_grad=True, device=device)
    outputs = model(input).clone()
    loss = outputs.sum().clone()
    loss.backward()
</code></pre>
<p>I get an error:</p>
<pre data-code-wrap="bash"><code class="lang-bash">Traceback (most recent call last):
  File "runme.py", line 15, in &lt;module&gt;
    loss.backward()
  File "...lib/python3.12/site-packages/torch/_tensor.py", line 626, in backward
    torch.autograd.backward(
  File "...lib/python3.12/site-packages/torch/autograd/__init__.py", line 347, in backward
    _engine_run_backward(
  File "...lib/python3.12/site-packages/torch/autograd/graph.py", line 823, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Error: accessing tensor output of CUDAGraphs that has been overwritten by a subsequent run. Stack trace: File "...lib/python3.12/site-packages/torch/_dynamo/external_utils.py", line 45, in inner
    return fn(*args, **kwargs). To prevent overwriting, clone the tensor outside of torch.compile() or call torch.compiler.cudagraph_mark_step_begin() before each model invocation.
</code></pre>
<p>Im using torch ‘2.6.0+cu124’, CUDA Version 12.4, Ubuntu 22.04.5, with NVIDIA A10G.<br>
Any help appreciated</p>
Question about CPU memory usage during training

viet2411: <p>Hi guys,<br>
I’m training my model using pytorch.<br>
I created a simple neural network with 2 layers training on MNIST dataset, and applied a custom method named <strong>LS</strong> on every neuron between two hidden layers.<br>
However, during the training, my memory usage constantly increased over time so I’m running out of memory and the process was killed in the first epoch.<br>
Below is my implementation of the neural network and the training process:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(28*28*1, 20)
        self.fc2 = nn.Linear(20, 10) 
    def forward(self, x):
        x1 = self.fc1(x)
        x2 = torch.sigmoid(x1)
        x3 = self.fc2(x2)
        com_x3 = torch.sigmoid(x3)
        x2 = x2.clone().requires_grad_()
        for i in range(x2.shape[0]):
            for j in range(x2.shape[1]):
                for k in range(com_x3.shape[1]):
                    # Applying
                    x2[i,j] = LS(x2[i,j],com_x3[i,k])
        return x3

def train(epoch):
    train_loss = 0  
    train_accuracy = 0  
    model.train()
    for data, label in tqdm(loader_train, desc="Training"):
        data, label = data.view(-1, 28*28).to(device), label.to(device)
        optimizer.zero_grad()  
        y_pred_prob = model(data)  
        torch.autograd.set_detect_anomaly(True)
        loss = loss_fn(y_pred_prob, label)  
        loss.backward()  
        optimizer.step()  
        train_loss += loss.item()
        y_pred_label = torch.max(y_pred_prob, 1)[1]
        train_accuracy += torch.sum(y_pred_label == label).item() / len(label)
</code></pre>
<p>I’m using pytorch 2.6, running the code on WSL2 (RAM for WSL is 24Gb). Batch size is 128.<br>
Please help me with this problem.<br>
Thank you and best regards.</p>

hansc: <p>Unrelated to your memory issues, but do you mean to return <code>x2</code> instead of <code>x3</code> at the end of your forward pass?</p>

hansc: <p>Also the triple nested for loop seems like a red flag here, I’d recommend trying to vectorize this but without knowing what LS is I won’t jump to that conclusion.</p>

viet2411: <p>Here’s my LS method:</p>
<pre><code class="lang-auto">def LS(x, y):
    ls_a = x
    ls_b = 1.0 - x
    ls_c = 1.0 - y
    ls_d = y
    
    # Original conditions
    if ls_a == 0.0: ls_a = 1e-03
    if ls_b == 0.0: ls_b = 1e-03
    if ls_c == 0.0: ls_c = 1e-03
    if ls_d == 0.0: ls_d = 1e-03
 
    term_a = (ls_b * ls_d) / (ls_b + ls_d)
    term_b = (ls_a * ls_c) / (ls_a + ls_c)
    term_c = (ls_b * ls_d) / (ls_b + ls_d)
    
    # Original conditions
    if term_a == 0.0: term_a = 1e-03
    if term_b == 0.0: term_b = 1e-03
    if term_c == 0.0: term_c = 1e-03

    ls_res = (ls_a + term_a) / (ls_a + ls_b + term_b + term_c)
    return ls_res
</code></pre>
<p>I want to update x2 by LS, and use it during the backward pass.</p>
RTX 5080 compatibility issues with stable diffusion

xlab: <p>I know this has already been answered but I’m going to ask one more time.</p>
<p>I’ve got this type of error:</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
</code></pre>
<p>Or the sm120 thing.</p>
<p>And Yes I already downloaded the nightly version 12.8.<br>
Before that I’ve had completely removed the previous version (I’ll add a screen of the new installation in case I missed<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a143607b9642dd3dfb66bba69363d0912da443f3.png" data-download-href="https://discuss.pytorch.org/uploads/default/a143607b9642dd3dfb66bba69363d0912da443f3" title="Capture d'écran 2025-03-30 040620"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/1/a143607b9642dd3dfb66bba69363d0912da443f3.png" alt="Capture d'écran 2025-03-30 040620" data-base62-sha1="n0Be4NpfghRvWS5zQLDl9wTMRCX" width="690" height="311" data-dominant-color="181817"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Capture d'écran 2025-03-30 040620</span><span class="informations">1912×862 66.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
something).</p>
<p>(Also uninstall and install webui, the model, python, everything but windows.)</p>
<p>I’m on windows 11 using python 3.10.9</p>
<p>I’m crying <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Try to narrow down which kernel fails. The nightly PyTorch binary built with CUDA 12.8 ships with Blackwell kernels so I guess a third party library might trigger this error.</p>
Will doing two times forward and backward work fine?

chener: <p>I recently find that code like the following doesn’t work well for ddp (broadcast_buffers=True)</p>
<pre data-code-wrap="python"><code class="lang-python">loss1 = L1(model(x1), y1)
loss2 = L1(model(x2), y2)
loss = loss1+loss2
loss.backward()
optimizer.step()
optimizer.zero_grad()
</code></pre>
<p>Someone recommend using another wrapper to do something like the following</p>
<pre><code class="lang-auto">class Wrapper():
    def __init__(self, model):
        self.model = model
    def forward(self, x1, x2, y1, y2):
        loss1 = L1(self.model(x1), y1)
        loss2 = L1(self.model(x2), y2)
        loss = loss1+loss2
        return loss

loss = wrapper(x1,x2,y1,y2)

loss.backward()
optimizer.step()
optimizer.zero_grad()
</code></pre>
<p>I don’t want to use the wrapper because my actual code is more complicated to implement a wrapper like the above.<br>
But if I change the code to</p>
<pre><code class="lang-auto"># L1 represents L1 loss; model represents my model
loss1 = L1(model(x1), y1)
loss1.backward()

loss2 = L1(model(x2), y2)
loss2.backward()

optimizer.step()
optimizer.zero_grad()
</code></pre>
<p>Does the code work equally to the code using  wrapper?</p>

KFrank: <p>Hi Chener!</p>
<aside class="quote no-group" data-username="chener" data-post="1" data-topic="218388">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/chener/48/17159_2.png" class="avatar"> chener:</div>
<blockquote>
<p>Does the code work equally to the code using wrapper?</p>
</blockquote>
</aside>
<p>Yes, your third version where you call <code>.backward()</code> twice, is mathematically the same<br>
as your first two versions where you sum the two losses and then call <code>.backward()</code><br>
once.  (It could well have slightly different numerical round-off error.)</p>
<p>When you call <code>.backward()</code> twice (with no intervening <code>.zero_grad()</code>) the two gradients<br>
get accumulated into the <code>.grad</code> properties of the various parameters.  But the sum of the<br>
gradients is the gradient of the sum, so you get the same result as summing the losses and<br>
calculating the gradient of the sum with a single call to <code>.backward()</code>.</p>
<p>Depending on your use case and the amount of memory you have, you could combine<br>
your inputs, <code>x1</code> and <code>x2</code>, into a single batch tensor, <code>x_both</code>, and, likewise, combine<br>
your targets into a single batch tensor, <code>y_both</code>, and perform a single forward and<br>
backward pass:</p>
<pre data-code-wrap="python"><code class="lang-python">loss_both = L1 (model (x_both), y_both)
loss_both.backward()
optimizer.step()
optimizer.zero_grad()
</code></pre>
<p>Assuming that your loss function, <code>L1()</code>, uses (the equivalent of) <code>reduction = 'sum'</code><br>
(or nearly equivalently, <code>reduction = 'mean'</code>), this will be mathematically equivalent<br>
to your three versions.</p>
<p>Best.</p>
<p>K. Frank</p>
Flex_attention inference with variable length inputs

jjjj: <p>I’m using flex_attention (or scaled_dot_product_attention).</p>
<p>At inference time the length of the sequence can vary (but the mask is otherwise not data dependent).</p>
<p>When not using flex_attention I can just trim the mask to the sequence length (inference is batch size 1)</p>
<p>mask = dense_mask[:, :, (S - L) : S, :S]</p>
<p>(and if I enable capture scalars this even plays nice with torch.compile).</p>
<p>I understand recomputing the block attention mask is somewhat expensive. Is there a recommended pathway to trim the block attention mask at inference time that can avoid need to recompute it?</p>

windsornguyen: <p>This thread might be related! <a href="https://github.com/pytorch-labs/attention-gym/issues/109" class="inline-onebox" rel="noopener nofollow ugc">Dynamic mask block sizes during inference · Issue #109 · pytorch-labs/attention-gym · GitHub</a></p>
GAN model-Loss selection for a beginner

Li-Yun: <p>Hello, I am new to GANs. I aim at building a GAN model to start generating Indoor scenes (of size 128x128 or 256x256). I have a quite big dataset (more that 30k images).</p>
<p>I gave a try with DCGAN in PyTorch framework, using this implementation (<a href="https://github.com/eriklindernoren/PyTorch-GAN/blob/master/implementations/dcgan/dcgan.py" rel="nofollow noopener">https://github.com/eriklindernoren/PyTorch-GAN/blob/master/implementations/dcgan/dcgan.py</a>) with BCE loss, but didn’t manage to generate even a structure “close” to the desired target distribution.</p>
<p>Due to the fact that DCGAN is an old architecture (noticed that it uses upsampling instead of transposed convolutions, and Is used mostly for image sizes of 32x32). Could someone please give me some advice on a good starting point for my model (any PyTorch repo with GAN architecture and loss) so that I can further build my implementation?*</p>

overmach: <p>Hello, did you make progresses about loss function selection?</p>
Why do we have two separated posts "Custom C++ and CUDA Operators" and "Custom C++ and CUDA Extensions"

Aakira: <p>Hi, I found two different articles in the PyTorch documentation: <em><a href="https://pytorch.org/tutorials/advanced/cpp_custom_ops.html" rel="noopener nofollow ugc">Custom C++ and CUDA Operators</a></em> and <em><a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">Custom C++ and CUDA Extensions</a></em>. After reading through them, I noticed that (if I understand correctly) <em>Custom C++ and CUDA Extensions</em> already covers the content of <em>Custom C++ and CUDA Operators</em> while also providing a much simpler syntax.</p>
<p>However, I don’t quite understand why these two articles are separated. Is there any other purpose that I might have missed while reading?</p>

ptrblck: <p>The extensions tutorial was written in 2018 describing how to add any custom extension to PyTorch and how to call it (in Eager mode). The operator tutorial was written in 2024 and describes how to let <code>torch.compile</code> work with your newly added operator and is also part of the <code>torch.library</code> docs.<br>
I agree both articles are quite similar and I would guess might be cleaned up to make clear when to use which approach.</p>
Issues with Conv2D Formula for Padding in Integer-Arithmetic-Only Inference

xiechengyan: <p>Hi,<br>
I am currently working on implementing an FPGA to accelerate the inference of a quantized model using Integer-Arithmetic-Only Inference, as described in <a href="https://arxiv.org/abs/1712.05877" rel="noopener nofollow ugc">this paper</a>.</p>
<p>However, the issue I’m facing is that in a Conv2D layer with padding, the formula below seems to be incorrect for calculations.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/b/eb09db80a2f24b5a79f58c446a0135d75d5b4212.png" data-download-href="https://discuss.pytorch.org/uploads/default/eb09db80a2f24b5a79f58c446a0135d75d5b4212" title=""><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/b/eb09db80a2f24b5a79f58c446a0135d75d5b4212.png" alt="" data-base62-sha1="xxfk9jRxh3REnIRrRsbOUdehRPI" width="440" height="154" role="presentation"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename"></span><span class="informations">440×154 5.35 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Could someone provide the correct formula? Thanks!</p>
GQA support in scaled_dot_product_attention

mseeger: <p>In the docs, I see something about GQA support, allowing <code>number_of_heads_key_value &lt; number_of_heads_query</code>. When I call it this way (e.g., <code>number_of_heads_key_value == 1, number_of_heads_query &gt; 1</code>, I am getting an exception. I have torch==2.5.1.</p>
<p>Does anybody know when this will be done? It is important to support not only GQA, but also multi-head latent attention for the DeepSeek models. For inference, I do not need this and can write it myself (<code>matmul</code> supports broadcasting in the batch dimensions), but when I want to fine-tune such models, I’d really need FlashAttention etc., and with this feature missing, I am stuck.</p>

mseeger: <p>Whoever is behind this implementation: The case I am mentioning here, is very relevant for DeepSeek (multi-head latent attention), and such an obvious way to save memory and maybe GPU compute. For example, if I run DeepSeek with 128 heads, the inputs can be smaller by a factor of 128.</p>

ReadyPlayerEmma: <p>I would also really appreciate GQA support for MPS. Is this a driver/feature limitation with Apple’s MPS internal matmul/attention kernels?</p>
Loaded checkpoint giving constant output during validation

anirudh_puligandla: <p>I am loading a model checkpoint to evaluate it on the MNIST dataset. The checkpoint is loading correctly and producing correct outputs, loss, etc., when I resume the training.</p>
<p>In another script I load the checkpoint the same way to evaluate on the same data. However, the loaded model produces a constant prediction output for every model(input) call.</p>
<p>Am I missing something that automatically gets called during training? I cannot understand what am I missing here.</p>
<p>I load the checkpoint like this in both the scripts:</p>
<pre><code class="lang-auto">checkpoint = torch.load('file_name', weights_only=True)
model.load_state_dict(checkpoint['model_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
epoch = checkpoint['epoch']
loss = checkpoint['loss']
</code></pre>
<p>In the evaluation evaluation script I don’t load the optimizer, epoch and loss.<br>
I call “model.eval()” before evaluation.<br>
In the resume training script, I make these additional calls, that I feel are the only difference between the two scripts:</p>
<pre><code class="lang-auto">best_model = model.state_dict()
model.train(True)
</code></pre>
<p>I checked the layers’ weights after loading the checkpoint and they are the same in both the scripts.</p>

Alex_17127: <p>same issue and i can’t understand what’s going on</p>
Homomorphic encryption of tensors?

Geremia: <p>I thought <a href="https://crypten.ai/" rel="noopener nofollow ugc">CrypTen</a> implemented homomorphic encryption of tensors, but it’s a planned feature for the future. CrypTen is currently more for secure multiparty computation (MPC).</p>
<p>Is there another project that allows for homomorphic encryption of tensors such that <a href="https://discuss.pytorch.org/t/pytorch-compatible-gpu-rental/218132/3">cloud services</a> can train on encrypted tensors?</p>
Distributed Tutorial works with mp.spawn but not torchrun

Marcelo_Sena: <p>Hi, I am following the distributed data parallel tutorial for training on mulitple gpus. I am using an HPC server to run codes, requesting 2 gpus in a single node.</p>
<p>I have managed to succesfully run the code in the distributed tutorial here <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.6.0+cu124 documentation</a> with the mp.spawn syntax.</p>
<p>However, I get an error when I initialize it with torchrun, getting the error below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38bbe773a664165bad26af95d32fdf354409be61.png" data-download-href="https://discuss.pytorch.org/uploads/default/38bbe773a664165bad26af95d32fdf354409be61" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38bbe773a664165bad26af95d32fdf354409be61.png" alt="image" data-base62-sha1="85TjU50bJPJg7Hclvnj2BEfX87T" width="690" height="218" data-dominant-color="191919"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">944×299 13.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Here is the code I use to run it:<br>
torchrun --nnodes=1 --nproc_per_node=2 --standalone distributed_tutorial_script.py</p>
<p>Any help, hints or pointers are very much appreciated.<br>
Thanks!</p>
[Solved] Maskrcnn_benchmark + sm_90 (or Cuda > 11.8 or torch > 1.10 )

Priyank_Pathak: <p>So I have working maskrcnn_benchmark working with torch==1.10.0+cu111.</p>
<p>Now I shift to NVIDIA H100 which are sm_90.</p>
<p>sm_90 requires CUDA 11.8 which is supported on torch 1.13.0 onwards.<br>
torch 1.13.0 (or any other torch &gt;1.11) have removed &lt;THC/THC.h&gt; which mean I have to patch the<br>
<code>maskrcnn_benchmark/csrc/cuda/</code> with <code>&lt;ATen/ATen.h&gt;</code> following steps here : <a href="https://github.com/pytorch/pytorch/issues/72807" class="inline-onebox" rel="noopener nofollow ugc">Missing headers in ATen/cuda/DeviceUtils.cuh · Issue #72807 · pytorch/pytorch · GitHub</a></p>
<p>But once I do all of this I get</p>
<pre><code class="lang-auto">maskrcnn_benchmark/csrc/cuda/deform_pool_kernel_cuda.cu(245): error: no instance of overloaded function "atomicAdd" matches
 the argument list                                                                                                                                                                   
            argument types are: (c10::Half *, c10::Half)                                                                                                                             
          detected during instantiation of "void DeformablePSROIPoolBackwardAccKernel(int, const scalar_t *, const scalar_t *, int, scalar_t, int, int, int, int, int, int, scalar_t 
*, scalar_t *, const scalar_t *, const scalar_t *, const scalar_t *, int, scalar_t, int, int, int, int, int) [with scalar_t=c10::Half]"                                              
(347): here                                                
</code></pre>
<p>What do I do now <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>

Priyank_Pathak: <p>[UPDATE] After some chatgpt</p>
<pre><code class="lang-auto">
#include &lt;cuda_fp16.h&gt;
#include &lt;c10/util/Half.h&gt;

template &lt;typename scalar_t&gt;
__device__ void atomicAddHalf(scalar_t* address, scalar_t val) {
    if constexpr (std::is_same&lt;scalar_t, double&gt;::value) {
        atomicAdd(reinterpret_cast&lt;__half*&gt;(address), __double2half(val));
    } else if constexpr (std::is_same&lt;scalar_t, float&gt;::value) {
        atomicAdd(reinterpret_cast&lt;__half*&gt;(address), __float2half(val));
    } else {
        atomicAdd(reinterpret_cast&lt;__half*&gt;(address), static_cast&lt;__half&gt;(val));
    }
}
</code></pre>
<p>replace <code>atomicAdd</code> with <code>atomicAddHalf</code></p>
<p>Also in <code>setup.py</code></p>
<pre><code class="lang-auto">extra_compile_args["nvcc"] = [
            '-arch=sm_90',
            "-DCUDA_HAS_FP16=1",
            "-D__CUDA_NO_HALF_OPERATORS__",
            "-D__CUDA_NO_HALF_CONVERSIONS__",
            "-D__CUDA_NO_HALF2_OPERATORS__",
        ]
</code></pre>
<p>The build is successful. Now the code is compatible with even torch 2.4 as well which doesn’t require explicit <code>'-arch=sm_90'</code> argument</p>
How to optimize pytorch CNN with complex-numbers (no gradient used)?

Etienne_Guichard: <p>Hi,<br>
A basic recap of what we are doing. My team and I are building a Cellular Automata program (effectively Convolutional Networks) in PyTorch. Because we are not training them via backprop, we don’t need or use the gradients; they are fully disabled. Our kernels are quite large, so we use the FFT trick to speed up computation (we have tested it, and it is indeed faster than standard convolution). Our “image” sizes are in the range of <code>H, W = 500 -100</code>, <code>C = 1-3</code>, and<code> Batch = 1- 100</code>.</p>
<p>After the convolution, we pass the results to a custom forward function. The tensors the convolution produces are of size <code>(Batch, C, C, H, W)</code>, and they are multiplied with tensors that are representations of Fourier series with a number of harmonics <code>Num_harmonics</code> in the shape of <code>(Batch, Num_harmonics, C, C,1,1)</code>. Effectively, we apply a custom function constructed from the Fourier series to every pixel resulting from the convolution.</p>
<p>The custom forward function then looks like this:</p>
<p><code>growth = self.coefficients * torch.exp(1j * 2 * self.harmonics] * torch.pi * (x - self.shifts) /self.period),dim=1).real</code></p>
<p>Where,<code> self.coefficients</code>, <code>sel.harmonics</code>, <code>self.shifts</code>, and <code> self.periods</code> are batched tensors, and <code>x</code>  is the results from the convolution that have been expanded to meet the shape of the Fourier series tensors.</p>
<p>The issue is that batch sizes above 16 become quite slow. We know it is the equation above that is causing the problems, as we have tested it with other equations, such as:</p>
<p><code>growth = lambda x: 2 * torch.exp(-((x - mu) ** 2 / (sigma) ** 2) / 2) - 1</code></p>
<p>where <code>x</code>,<code> mu</code>, and <code>sigma</code> are batched tensors.</p>
<p>The equation above gives results that are about five times faster. Thus, it seems like the complex numbers are causing a slowdown.</p>
<p>We are using an RTX 4070 and are nowhere near the memory cap (about 3/12 GB). Everything is always on the GPU; we never transfer back to the CPU. Are there any tricks we can use to speed up the computation, like Channels Last Memory Format for FFT or reshaping the tensors?</p>
<p>Kind regards,<br>
Etienne</p>
Handling of multiple losses in inverse PINNs?

Mathieu: <p>I am working on a physics-informed-neural-network (PINN).</p>
<p>In the forward PINN (with fixed material parameter p = p_ref, input: spatial coordinates, output: pde field), the total loss represents the minimal energy stored in the system and the norm of the pde-residual.</p>
<blockquote>
<p>L_total = L_energy + L_residual</p>
</blockquote>
<p>This converges, regardless of the weights and bias initialization, to a fixed loss value (minimal energy) and L_residual = 0.</p>
<p>In the inverse PINNs, the idea is to train also the material parameter p (as a torch.nn.Parameter()) and adding an additional data loss term:</p>
<blockquote>
<p>L_total = L_energy + L_residual + L_data</p>
</blockquote>
<p>Ideally, the forward loss is recovered which is the case when the material parameter is optimized towards the ground truth value (p = p_ref → L_data = 0). However, the flexibility of training both the field and the material parameter can create a different minimum where the total inverse loss is lower than the total forward loss. For instance, there is a combination of “field+param” resulting in a not perfectly zero value of L_data, but a way lower value of L_energy, overall resulting in a lower total loss. So identification fails.</p>
<p>If I did a standard gradient-based optimization (e.g. Levenberg-Marquardt), I would only optimize the material parameter and solve a pde at every iteration to determine the field which has the minimal energy and L_residual = 0. The parameters can be re-identified.</p>
<p>Any advice on how to balance physics and data losses better—e.g., dynamic weighting, scaling strategies, or regularization—would be much appreciated.</p>
Torch "Dataset" class quirks?

sul: <p>Im having a hard time understanding the torch.utils.data.Dataset class, it doesn’t behave like a regular python class. I put below a simple example (output results are commented). In short:</p>
<ul>
<li>Dataset cannot access custom methods or variables regularly (see TestDataset)</li>
<li>But a class that inherits from Dataset can ! (see TestDataset2)<br>
Is there something im missing (static methods,…). Thanks in advance.</li>
</ul>
<pre><code class="lang-auto">import torch
from torch.utils.data import Dataset


# A basic python class
class TestObject():
    def __init__(self, a):
        self.a = a
    def get_a(self):
        return self.a
# print("testobject")
# testobject=TestObject(99)
# print(list(testobject.__dict__.keys()))# ['a']
# print(testobject.a)# 99
# print(testobject.get_a())# 99 

# A torch "Dataset" class: cant access attributes/methods normally
class TestDataset(Dataset):
    def __init__(self, a):
        self.a = a
    def get_a(self):
        return self.a
    def __len__(self):
        return 1
    def __getitem__(self, idx):
        return torch.randn(1,1,1)
# print("testdataset")
# testdataset=TestDataset(99)
# print(list(testdataset.__dict__.keys()))# ['a']
# print(testdataset.a)# AttributeError: type object 'TestDataset' has no attribute 'a'
# print(testdataset.get_a())# TypeError: TestDataset.get_a() missing 1 required positional argument: 'self'

## A basic python inherited class
class TestObject2(TestObject):
    def __init__(self, a,b):
        super().__init__(a)
        self.b = b
    def get_b(self):
        return self.b
# print("testobject2")
# testobject2=TestObject2(99,42)
# print(testobject2.a)# 99
# print(testobject2.b)# 42

# A torch "Dataset" inherited class: now everything works normally !?
class TestDataset2(TestDataset):
    def __init__(self, a,b):
        super().__init__(a)
        self.b = b
    def get_b(self):
        return self.b
# print("testdataset2")
# testdataset2=TestDataset2(99,42)
# print(list(testdataset2.__dict__.keys()))# ['a','b']
# print(testdataset2.a)# 99, it works now !?!
# print(testdataset2.get_a())# 99
# print(testdataset2.b)# 42
# print(testdataset2.get_b())# 42
</code></pre>

ptrblck: <p>I don’t get any <code>AttributeError</code> or <code>TypeError</code> running your code snippet and get:</p>
<pre data-code-wrap="python"><code class="lang-python">testobject
['a']
99
99
testdataset
['a']
99
99
testobject2
99
42
testdataset2
['a', 'b']
99
99
42
42
</code></pre>

sul: <p>Thanks for the quick reply! Well i feel stupid, it works after cleaning up my notebook and retesting. I probably had a tipo or just named something “Dataset”.</p>
What could be the reason for Model to lose all progress when put into .eval() mode?

Oliver: <p>Hi</p>
<p>I trained a GAN as my first PyTorch project and it worked really nicely.<br>
However, whenever I put the generator into .eval() mode under a torch.no_grad() wrapper to get some outputs I just get a uniform output like in an untrained model.</p>
<p>When I do the same without calling generator .eval() first it works fine.</p>
<p>I thought .eval() just makes sure, dropout, batch norm, etc are all in evaluation mode, so I don’t see how it would affect it other than I misunderstood what .eval() does.</p>
<p>Thanks in advance for any tips and thoughts of things I could try.<br>
It’s no biggie other than that I want to understand pytorch better.</p>

ptrblck: <aside class="quote no-group" data-username="Oliver" data-post="1" data-topic="138224">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/oliver/48/30054_2.png" class="avatar"> Oliver:</div>
<blockquote>
<p>I thought .eval() just makes sure, dropout, batch norm, etc are all in evaluation mode, so I don’t see how it would affect it other than I misunderstood what .eval() does.</p>
</blockquote>
</aside>
<p>Yes, your assumption is right and calling <code>model.eval()</code> will set all modules into evaluation mode. E.g. batchnorm layers will use their running stats to normalize the input activations instead of the current batch stats. Depending on how well these running stats were fit to your dataset (and assuming they can even converge towards the “dataset mean and var”) this approach could work well or fail (as is seen in other posts here).</p>

Oliver: <p>Oh, thank you so much! So, I’m guessing what I’m seeing is the mean of the training data somewhat. Hmmm. Interesting! Thanks again!</p>

overmach: <p>I am having the same issue.<br>
Did you find any training strategies that helped overcome it?</p>
Questions about linter for PyTorch

shaoyu_young: <p>Hi, guys.<br>
I am currently trying to find some linters to do some static analysis for pt, in order to find some <code>memory issues</code>.</p>
<p>Are there any off-the-shelf static analysis tools we can use (both check Python, CPP, and cuda)? My check includes <code>array overreach</code>, <code>memory access is not aligned</code> and so on.</p>
<p>thanks in advance!</p>

Soumya_Kundu: <aside class="onebox githubrepo" data-onebox-src="https://github.com/pytorch-labs/torchfix">
  <header class="source">

      <a href="https://github.com/pytorch-labs/torchfix" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <div class="github-row" data-github-private-repo="false">
  <img width="690" height="344" src="https://opengraph.githubassets.com/a8e1a1e46eb4987df5e05193636be432/pytorch-labs/torchfix" class="thumbnail">

  <h3><a href="https://github.com/pytorch-labs/torchfix" target="_blank" rel="noopener nofollow ugc">GitHub - pytorch-labs/torchfix: TorchFix - a linter for PyTorch-using code with...</a></h3>

    <p><span class="github-repo-description">TorchFix - a linter for PyTorch-using code with autofix support</span></p>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Maybe a start?</p>

shaoyu_young: <aside class="quote no-group quote-modified" data-username="Soumya_Kundu" data-post="2" data-topic="217769">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/soumya_kundu/48/71716_2.png" class="avatar"> Soumya_Kundu:</div>
<blockquote>
<p><a href="https://github.com/pytorch-labs/torchfix" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch-labs/torchfix: TorchFix - a linter for PyTorch-using code with autofix support</a></p>
</blockquote>
</aside>
<p>Thanks!<br>
It looks good. However, <code>torchfix</code> seems a <code>python</code> linter bug can’t detect any C++/CUDA problem? Is there any C++/CUDA linters which are used in PyTorch CI?</p>

ptrblck: <p>Yes, CI runs linting tests and will fail. Check out the <a href="https://github.com/pytorch/pytorch/blob/main/CONTRIBUTING.md#pre-commit-tidylinting-hook">contribution guide</a> for more information.</p>
How combined loss impact on separated network

Y_Z: <p>Assume that i have three separated classifiers connected to the same nodes (see the attached image). How will the .backward(), .step() impact the training in the following scenarios?</p>
<p>scenario 1:<br>
loss_1.backward()<br>
loss_1.step()<br>
loss_2.backward()<br>
loss_2.step()<br>
loss_3.backward()<br>
loss_3.step()</p>
<p>scenario 2:<br>
loss_1.backward()<br>
loss_2.backward()<br>
loss_3.backward()<br>
total_loss = loss_1 + loss_2 + loss_3<br>
total_loss.step()</p>
<p>scenario 3:<br>
total_loss = loss_1 + loss_2 + loss_3<br>
total_loss.backward()<br>
total_loss.step()</p>
<p>I’m wondering if I sum up the total loss from two separated classifiers, and backpropogate, will these separated classifiers only be influenced by their own part of the loss within the total_loss? Will  pytorch  automatically use each of these part within the total_loss, or just backpro based on the total_loss ?</p>
<p>Many thanks in advanced!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/0/10a9b99e8416315211cca8ec0d1c07a564dc4bd4.png" data-download-href="https://discuss.pytorch.org/uploads/default/10a9b99e8416315211cca8ec0d1c07a564dc4bd4" title="Screenshot_pytorch"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/0/10a9b99e8416315211cca8ec0d1c07a564dc4bd4.png" alt="Screenshot_pytorch" data-base62-sha1="2npgYzLAnS7ryMJMo41PKYUbzrm" width="346" height="313"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_pytorch</span><span class="informations">346×313 5.06 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><code>loss.step()</code> is not a valid call since you would need to call <code>optimizer.step()</code>, which contains the parameters it will update.</p>
<aside class="quote no-group" data-username="Y_Z" data-post="1" data-topic="218249">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/y_z/48/76336_2.png" class="avatar"> Y_Z:</div>
<blockquote>
<p>Will pytorch automatically use each of these part within the total_loss, or just backpro based on the total_loss ?</p>
</blockquote>
</aside>
<p>The <code>total_loss</code> is a sum of the sub-losses and will thus create a new computation node in your graph. Calling <code>total_loss.backward()</code> will then backpropagate through each classifier computing the gradients of their used parameters w.r.t. its loss.</p>

Y_Z: <p>Thanks for the reply!</p>
Double execution of __getitem__ on batch_size=1?

roach: <p>Hello, i was debugging my dataset and i observed that it invokes __getitem__ twice when batch size is set to 1. Is this normal behavior? (version torch==2.7.0.dev20250228)</p>
<pre><code class="lang-auto">    def __getitem__(self, idx):
        image = np.float32(load(self.img_file_locs[idx]).get_fdata())
        l_targ = self.img_file_locs[idx].split('_')[-3]
        target = np.float32(load(self.targets[l_targ]).get_fdata())

        normalized_image = (image - np.min(image)) / (np.max(image) - np.min(image))
        normalized_target = (target - np.min(target)) / (np.max(target) - np.min(target))
        image = torch.from_numpy(normalized_image).unsqueeze(0)
        target = torch.from_numpy(normalized_target).unsqueeze(0)

        if self.debug:
            print(f"image: {self.img_file_locs[idx]}")
            print(f"target: {self.targets[l_targ]}")

        return {'image': image, 'target': target}
</code></pre>
<p>Then trying it out:</p>
<pre><code class="lang-auto">rq_data = RatioDataset(tar_dir,rat_dir, use_glob=True, debug=True)
dataloader = DataLoader(rq_data, batch_size=1, num_workers=1)
q = next(iter(dataloader))
for _,v in q.items():
    print(v.shape)
</code></pre>
<p>Which outputs this:</p>
<pre><code class="lang-auto">image: /Users/petros/Documents/PhD_outputs/IEC_Simulated_Images/10s/r10/recon_10s_r10_002_corr.nii.gz
target: /Users/petros/Documents/PhD_outputs/activity_conc_images_from_cont_label/activity_image_R10_noBlur.nii
image: /Users/petros/Documents/PhD_outputs/IEC_Simulated_Images/5s/r2/recon_5s_r2_021_corr.nii.gz
target: /Users/petros/Documents/PhD_outputs/activity_conc_images_from_cont_label/activity_image_R2_noBlur.nii
torch.Size([1, 1, 256, 256, 71])
torch.Size([1, 1, 256, 256, 71])
</code></pre>
<p>Is this normal behavior?</p>
Torchtune distributed issue

donparker1: <p>I am trying to run torchtune on two nodes using the following command:</p>
<p><code>tune run --rdzv-endpoint "${M_ADDR}:1234" --nnodes 2 --nproc-per-node 8 --rdzv-id 102 --rdzv-backend=c10d lora_finetune_distributed --config recipes/configs/llama3/70B_full.yaml</code></p>
<p>however, I receive torch.distributed.elastic.rendezvous.api.RendezvousTimeoutError</p>
<p>Can someone confirm if the command is correct? Also, any suggestions on how to resolve issue ? Any tests I can try to help ?</p>

H-Huang: <p>I’m not sure about torchtune, but it seems like it is using torchrun: <a href="https://pytorch.org/docs/stable/elastic/run.html" class="inline-onebox" rel="noopener nofollow ugc">torchrun (Elastic Launch) — PyTorch 2.6 documentation</a></p>
<p>I’m guessing the M_ADDR you are using is not accessible. If you just torchrun and a simple script does that work?</p>