akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
cuFFT half precision error using rfft2/irfft2

Mole_Turner: <p>I’m using Automatic Mixed precision and within my model’s forward pass I make use of <code>torch.fft.rfft2()</code> and <code>torch.fft.irfft2()</code>. For some reason this seems to work find on my laptop, but when using a remote node via slurm throws the following error</p>
<p><code>RuntimeError: cuFFT only supports dimensions whose sizes are powers of two when computing in half precision</code></p>
<p>As far as I can tell I’m using the same versions of pytorch on each</p>
<p><code>Remote:</code></p>
<p><code>pytorch                    2.5.1            py3.12_cuda12.4_cudnn9.1.0_0  pytorch</code></p>
<p><code>pytorch-cuda               12.4             hc786d27_7                    pytorch</code></p>
<p>Laptop:</p>
<p><code>pytorch                   2.5.1           py3.12_cuda12.4_cudnn9_0    pytorch</code></p>
<p><code>pytorch-cuda              12.4                 h3fd98bf_7    pytorch</code></p>
<p>I could just increase the padding to next power of 2….however this drastically increases the size of the two matrices I’m then looking to multiply together.</p>

ptrblck: <p>Do you see the same error using the latest stable release?</p>

Mole_Turner: <p>I haven’t had chance to check that yet, as I need to create a new env on the remote node. However, I have found the latest(?) cuFFT documentation <a href="https://docs.nvidia.com/cuda/pdf/CUFFT_Library.pdf" rel="noopener nofollow ugc">here</a>, which states the following</p>
<blockquote>
<p>Half-precision transforms have the following limitations:</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/play_button.png?v=14" title=":play_button:" class="emoji" alt=":play_button:" loading="lazy" width="20" height="20"> Minimum GPU architecture is SM_75</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/play_button.png?v=14" title=":play_button:" class="emoji" alt=":play_button:" loading="lazy" width="20" height="20"> Sizes are restricted to powers of two only</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/play_button.png?v=14" title=":play_button:" class="emoji" alt=":play_button:" loading="lazy" width="20" height="20"> Strides on the real part of real-to-complex and complex-to-real transforms are not supported <img src="https://discuss.pytorch.org/images/emoji/apple/play_button.png?v=14" title=":play_button:" class="emoji" alt=":play_button:" loading="lazy" width="20" height="20"> More than one GPU is not supported</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/play_button.png?v=14" title=":play_button:" class="emoji" alt=":play_button:" loading="lazy" width="20" height="20"> Transforms spanning more than 4 billion elements are not supported</p>
</blockquote>
<p>This seems to be quite explicit that my issue is probably due to the second point, however, I’m now further confused as to why it seems to run fine on my laptop.</p>

ptrblck: <p>Are you sure you are using the GPU and cuFFT in your laptop environment? Could you profile the workload and check the kernel names?</p>
Buffer Overflow resolution

tm-kachi: <p>I saw that 2.7.0 had a Buffer Overflow vulnerability outlined here <a href="https://www.cvedetails.com/cve/CVE-2025-55558/" rel="noopener nofollow ugc">https://www.cvedetails.com/cve/CVE-2025-55558/</a>. Upgraded to 2.9.0 but vulnerability wasn’t fixed. Wondering if this would be resolved soon?</p>

ptrblck: <p>Is the CVE not already fixed via <a href="https://github.com/pytorch/pytorch/issues/151523">this</a> and <a href="https://github.com/pytorch/pytorch/pull/151887">this</a> PR as linked in the CVE description?</p>
Is torch.autocast using mixed precision for matmul?

Alessandro_Mammana: <p>Dear all,</p>
<p>I am trying to find out what happens when we do matmuls (for example with nn.Linear) within a block annotated with <code>torch.autocast(device_type="cuda", dtype=torch.float16):</code></p>
<p>e.g. let’s say I have:</p>
<pre><code class="lang-auto">x = torch.randn(4, 4).to("cuda")
m = nn.Linear(4, 16)

with torch.autocast(device_type="cuda", dtype=torch.float16):
    y = m(x)
    print(y.dtype)

</code></pre>
<p>What I would like to confirm is that the multiplications happen with fp16 but the accumulation is done using fp32. At least that’s what I expect from the docs and from <a href="https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html#amp" class="inline-onebox" rel="noopener nofollow ugc">Train With Mixed Precision - NVIDIA Docs</a> .</p>
<p>However when I run code like the one above I cant find any convincing evidence that the accumulation is happening with fp32. The ouput dtype printed above is fp16 and when I profile similar code with nsys I see the operation <code>ampere_fp16_s1688gemm_fp16_128x128_ldg8_f2f_stages_32x1_tn</code> which, at least by the name, doesn’t mention fp32 (and also the following operations take an fp16 as input without conversion in between).</p>
<p>Maybe <code>ampere_fp16_s1688gemm_fp16_128x128_ldg8_f2f_stages_32x1_tn</code>  is <strong>internally</strong> accumulating into an fp32 and then converting to fp16? How can I debug this further? I’d like to avoid a matmul with fp16 to accumulate the sums because I expect that to have numerical stability issues and it’s not what is advertised.</p>
<p>I am doing this profiling on NVIDIA L4 Tensor Core GPUs (amazon g6 instances) and I am using torch version 2.6.0+cu124</p>
Cannot reproduce grid_sample results

Andrea_Rosasco: <p>Hello, I’m trying to explicitly write down an equivalent to <code>torch.nn.functional.grid_sample()</code> the reason is that I need to build a TensorRT engine that executes it but the onnx node <code>GridSample</code> is not yet implemented in trt.</p>
<p>To start, I’ve tried re-implementing the <a href="https://github.com/torch/nn/blob/master/lib/THNN/generic/SpatialGridSamplerBilinear.c" rel="noopener nofollow ugc">c algorithm</a> in python:</p>
<pre><code class="lang-auto">def inside(x, y, H, W):
    return 0 &lt;= x &lt; W and 0 &lt;= y &lt; H


def grid_sample(img, coords, mode="bilinear", padding_mode="zeros", align_corners=False):
    N, C, IH, IW = img.shape
    H, W = coords.shape[1:3]

    output = torch.zeros([N, C, H, W])

    for n in range(N):
        for h in range(H):
            for w in range(W):
                ix = coords[n, h, w, 0]
                iy = coords[n, h, w, 1]

                ix = ((ix + 1) / 2) * (IW - 1)
                iy = ((iy + 1) / 2) * (IH - 1)

                ix_nw = floor(ix)
                iy_nw = floor(iy)
                ix_ne = ix_nw + 1
                iy_ne = iy_nw
                ix_sw = ix_nw
                iy_sw = iy_nw + 1
                ix_se = ix_nw + 1
                iy_se = iy_nw + 1

                nw = (ix_se - ix) * (iy_se - iy)
                ne = (ix - ix_sw) * (iy_sw - iy)
                sw = (ix_ne - ix) * (iy - iy_ne)
                se = (ix - ix_nw) * (iy - iy_nw)

                for c in range(C):
                    nw_val = img[n, c, iy_nw, ix_nw] if inside(ix_nw, iy_nw, IH, IW) else 0
                    ne_val = img[n, c, iy_ne, ix_ne] if inside(ix_ne, iy_ne, IH, IW) else 0
                    sw_val = img[n, c, iy_sw, ix_sw] if inside(ix_sw, iy_sw, IH, IW) else 0
                    se_val = img[n, c, iy_se, ix_se] if inside(ix_se, iy_se, IH, IW) else 0
                    out_val = nw_val * nw + ne_val * ne + sw_val * sw + se_val * se

                    output[n, c, h, w] = out_val

    return output
</code></pre>
<p>This is the result but even executing this I get different value than the one returned by <code>F.grid_sample(img, grid, mode="bilinear", padding_mode="zeros", align_corners=False)</code><br>
Why could that be?</p>

dina_abouzeid: <p>Hey, Did you find the solution? I’m also facing the same issue.<br>
Thanks in advance</p>
About some issue about torch.median()

pistachio: <p>when i want to use torch.median(), i found that the median of even number of data point is not as expected</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/0/402ab138c482f23fce3619f76d7819507cb92190.png" data-download-href="https://discuss.pytorch.org/uploads/default/402ab138c482f23fce3619f76d7819507cb92190" title="torch_median"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/402ab138c482f23fce3619f76d7819507cb92190_2_690x332.png" alt="torch_median" data-base62-sha1="99E1Bu4SEImQQsInzmKE6YVZPtm" width="690" height="332" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/402ab138c482f23fce3619f76d7819507cb92190_2_690x332.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/402ab138c482f23fce3619f76d7819507cb92190_2_1035x498.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/0/402ab138c482f23fce3619f76d7819507cb92190_2_1380x664.png 2x" data-dominant-color="333431"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">torch_median</span><span class="informations">1810×872 72.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>why col_medians is not tensor([2.5, 4.5, 3.5, 5.5])</p>
<p>the version of torch is  2.8.0+cu128.</p>

ptrblck: <p>From the <a href="https://docs.pytorch.org/docs/stable/generated/torch.median.html">docs</a>:</p>
<blockquote>
<p>The median is not unique for <code>input</code> tensors with an even number of elements in the dimension <code>dim</code>. In this case the lower of the two medians is returned. To compute the mean of both medians in <code>input</code>, use <a href="https://docs.pytorch.org/docs/stable/generated/torch.quantile.html#torch.quantile" title="torch.quantile"><code>torch.quantile()</code></a> with <code>q=0.5</code> instead.</p>
</blockquote>
Will only update to 2.6.0

Lexi_Palmer: <p>I’m trying to install and run vllm &amp; several of the dependencies require pytorch 2.8.0, however no matter what I try the most recent version I can install is 2.6.0 (updating only goes to 2.6 and after trying to specify version, pip returns there is no version 2.8). For various reasons building from source (per the git page) is not an option. Any suggestions appreciated!</p>

ptrblck: <p>Could you post your OS, Python version, CPU arch, please?</p>

Lexi_Palmer: <p>Yes sorry -</p>
<p>OS - CentOS Linux release 7.4.1708 (Core)</p>
<p>Python 3.11.4</p>
<p>CPU x86_64</p>

ptrblck: <p>Thank you!</p>
<p>I cannot reproduce any issues and can install PyTorch 2.8.0 with CUDA 12.8 using <code>pip install torch</code> as given in our <a href="https://pytorch.org/get-started/locally/">install instructions</a>.</p>
<pre><code class="lang-auto">lscpu | grep -i arch
Architecture:                    x86_64
python --version
Python 3.11.13
pip install torch 
Collecting torch
  Downloading torch-2.8.0-cp311-cp311-manylinux_2_28_x86_64.whl.metadata (30 kB)
...
</code></pre>
<p>I don’t know what might be causing the issue in your setup, but you could try to manually download the wheel and <code>pip install</code> it from your local folder. Maybe the error message would be better in this case.</p>

Lexi_Palmer: <p>Yeah - I have an old version of torch I can’t remove (working on an HPC so some files are blocked) but when I try to upgrade torch it defaults to 2.6.0:</p>
<pre><code class="lang-auto">pip install -U torch
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: torch in /share/apps/centos7/anaconda3/2023.07/lib/python3.11/site-packages (2.0.1)
Collecting torch
  Downloading torch-2.6.0-cp311-cp311-manylinux1_x86_64.whl.metadata (28 kB)
```
</code></pre>
In pytorch, why is grad_fn stored in next node, since it uses the previous nodes value to calculate grad and updates the grad in the previous node

liyuan: <p>In pytorch, why is grad_fn stored in next node and not in the previous node? I think it should be stored in the previous node, since it uses the previous nodes value to calculate grad and updates the grad in the previous node.</p>
<p>For example, suppose we have y = x**3</p>
<p>Then dy/dx = 3<em>x</em>*2</p>
<p>This is stored in the grad_fn of y. But it requires the value of the previous node, x, and is used to update the grad property of the previous node, x.</p>
<p>One possible argument I see is that x may be used in multiple separate paths. But in that case I don’t see why there cannot just be multiple grad_fn stored in x, and when updating just sum the grad from each grad_fn.</p>

KFrank: <p>Hi Liyuan!</p>
<aside class="quote no-group" data-username="liyuan" data-post="1" data-topic="223609" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/liyuan/48/78440_2.png" class="avatar"> liyuan:</div>
<blockquote>
<p>it uses the previous nodes value to calculate grad</p>
</blockquote>
</aside>
<p>This is not true in general.  (It’s probably true in your case.)</p>
<p>Consider:</p>
<p><code>y = a * x</code> (where <code>a.requires_grad = False</code>)</p>
<p>Here the tensor <code>a</code> is needed to compute <code>y.grad</code> (but neither <code>x</code> nor <code>y</code> is needed).</p>
<p><code>y = x**2</code></p>
<p>Here <code>x</code> is needed (more or less as in your example).</p>
<p><code>y = exp (x)</code></p>
<p>Here <code>y</code> is needed for <code>y.grad</code> (and <code>x</code> is not needed).</p>
<p>So the tensor(s) needed for backpropagation are often not naturally part of the “previous node.”</p>
<aside class="quote no-group">
<blockquote>
<p>and updates the grad in the previous node.</p>
</blockquote>
</aside>
<p>Let <code>x</code> and <code>y</code> both carry <code>requires_grad = True</code> and <code>z = x @ y</code>.</p>
<p>Backpropagation from <code>z</code> flows back to both <code>x</code> and <code>y</code>.  The <code>grad_fn</code> for the <code>z</code> node is<br>
<code>grad_fn=&lt;MmBackward0&gt;</code>.  Pytorch chooses to store <code>grad_fn</code> once in node <code>z</code>, rather<br>
than twice in both nodes <code>x</code> and <code>y</code>.</p>
<p>Last, I believe (but am not certain) that <code>grad_fn</code> contains the link(s) to the next (upstream)<br>
node(s) in the computation graph.  Since the computation graph is traversed from “result<br>
node” (downstream) to “argument node(s)” (upstream), backpropagation needs <code>grad_fn</code><br>
to be associated with “result node” in order to be able to find the “argument node(s).”</p>
<p>Best.</p>
<p>K. Frank</p>
Implementation of Hierarchical Actor Critic with PPolicy-on Policy-off Policy Optimization for primitive actions

Lordking1624: <p>Hello everyone,</p>
<p>I’m currently undertaking a Hierarchical Reinforcement Learning (<strong>HRL</strong>) project and I am attempting to implement a hybrid architecture based on the Hierarchical Actor-Critic (<strong>HAC</strong>) framework.</p>
<p>I started from the original <strong>HAC</strong> algorithm which uses <strong>DDPG</strong> and HER at all <strong>3 layers</strong>, then tried to replace the lowest-level which controls the <strong>primitive actions</strong> (e.g., joint torques in a robotic manipulator), with Policy-on Policy-off Policy Optimization (<strong>P3O</strong>) algorithm. (At first I tried to use <strong>PPO</strong> but <strong>P3O</strong> seemed to fit the off-policy architecture of the higher layers better)</p>
<p>I am using the environment found in the original <strong>HAC</strong> implementation: <code>https://github.com/andrew-j-levy/Hierarchical-Actor-Critc-HAC-</code></p>
<p>I have managed to integrate the <strong>P3O</strong> agent into the hierarchical loop. The overall system runs, policies are being sampled, and updates are being performed, but the agent is not learning any effective behavior. The high-level policies seem to struggle to set meaningful subgoals, and the low-level <strong>P3O</strong> agent never converges to a policy that reliably reaches the assigned subgoals.</p>
<p>I would like to know if anybody heard of something similar before, and/or if this architecture seems feasible and maybe useful for some cases.</p>
<p>Thank you in advance for any guidance.</p>
AOT-inductor run_impl uses a process-wide allocator: can two AOTIModelPackageLoader workers in the same process break each other’s CUDA Graphs?

jttzhang: <p>After inspecting the generated <code>xxx.wrapper.cpp</code> I see something like:</p>
<pre><code class="lang-auto">static constexpr int64_t int_array_1[] = {1L, };
AtenTensorHandle pool2_handle;
AOTI_TORCH_ERROR_CODE_CHECK(
    aoti_torch_empty_strided(
        1, int_array_0, int_array_1,
        cached_torch_dtype_uint8,
        cached_torch_device_type_cuda,
        this-&gt;device_idx_,
        &amp;pool2_handle));
RAIIAtenTensorHandle pool2(pool2_handle);
// … later alloc_from_pool calls …
pool2.reset();
</code></pre>
<p>After loader1 finishes capturing its CUDA graph, loader2 may still allocate the same virtual address. If I then run both loaders concurrently on two different streams, could this cause memory corruption or undefined behavior?</p>
UnpicklingError when loading a trained model

stoneMason101: <p>I have been trying to load a trained model.</p>
<p>This is my code block</p>
<pre><code class="lang-auto">import torch
from torch.serialization import add_safe_globals
import numpy as np

# If PyTorch complains about missing globals:
add_safe_globals([np.core.multiarray._reconstruct])
state_dict = torch.load("best.pt", weights_only=True, map_location="cpu")
</code></pre>
<p>I am getting below error message.</p>
<pre><code class="lang-auto">UnpicklingError: Weights only load failed. This file can still be loaded, to do so you have two options, do those steps only if you trust the source of the checkpoint. 
	(1) In PyTorch 2.6, we changed the default value of the `weights_only` argument in `torch.load` from `False` to `True`. Re-running `torch.load` with `weights_only` set to `False` will likely succeed, but it can result in arbitrary code execution. Do it only if you got the file from a trusted source.
	(2) Alternatively, to load with `weights_only=True` please check the recommended steps in the following error message.
	WeightsUnpickler error: Unsupported global: GLOBAL numpy.core.multiarray._reconstruct was not an allowed global by default. Please use `torch.serialization.add_safe_globals([numpy.core.multiarray._reconstruct])` or the `torch.serialization.safe_globals([numpy.core.multiarray._reconstruct])` context manager to allowlist this global if you trust this class/function.
</code></pre>
<p>Can you please help me to solve this please ?</p>
Torch.compile [WARNING] not enough SMs to use max_autotune_gemm mode

anass_017: <p>I have a small model with 900K in wights</p>
<p><code>compiled_model = torch.compile(net_model,backend="inductor",mode="max-autotune")</code></p>
<p>When I compile the model using max_autotune  model, he is giving me this warning :<br>
<code>  [WARNING] not enough SMs to use max_autotune_gemm mode and i don't know how to exploit thi feature.</code></p>
<p>PS: I’m working on complex- valued models  model with a ComplexConvolution class that uses two nn.Conv2D (for real and imaginary part )</p>

ptrblck: <p>The warning seems to be raised <a href="https://github.com/pytorch/pytorch/blob/e5f5bcf6d4ec022558caf4d0611d928497394a88/torch/_inductor/utils.py#L644-L645">here</a>, which assumes your GPU needs at least 80 SMs for this mode, which seems to be a hard-coded limitation in <code>torch.compile</code>.</p>

anass_017: <p>OK, i get it. I have only 48 streaming, thank you for your response.</p>

Prakhar: <p>Did you somehow resolve this and were able to use the max-autotune mode?</p>

mahimairaja: <p><a class="mention" href="/u/prakhar">@Prakhar</a> I think after providing enough GPU resource it should automatically fix</p>
Understanding reproducibility

Tugdual: <p>We are doing research in ML and wish to make sure that our experiments are reproducible - unfortunately we have over 100 different configurations to run and make sure that they converge to values that have been published.</p>
<p>What tools exist out there to make sure that code refactors, version changes etc. don’t affect previous performance negatively ?</p>
<p>How does pytorch/examples make sure of this ?</p>
Utilizing GPUs for a machine-translation model

Cuneyt_Tyler: <p>I am implementing RCTM I model proposed in famous paper “Recurrent Continuous Translation Models” which is a precursor to transformers. I have successfully built and trained it on very small datasets and get meaningful results. Now, I’d like to train it on a big dataset and utilize GPU (Multiple GPUs). I run it with my ‘cuda’ device on my local, but the GPU is old so it’s only for experimental results. When I run it on Kaggle with 2XT4 GPUs, i only have 20% GPU usage and when I try to train it with relatively big batches (e.g 1000) it basically fails to perform well. After 5 mins, I don’t even see any GPU usage in the status bar of Kaggle.</p>
<p>I move all of my tensors as well as my model to GPU device. I also use nn.DataParallel to tell torch to make use of multiple GPUs.</p>
<p>You can access my notebook here: <a href="https://www.kaggle.com/code/cuneyttyler/rctm-5" class="inline-onebox" rel="noopener nofollow ugc">rctm-5 | Kaggle</a></p>
<p>The model basically takes a sentence as a sequence of one-hot encoded vectors. It convolves it a number of times to have a final q x 1 vector in the end (q = representation size). This part is called Convolutional Sentence Model (CSM). This vector is used in a RNN layer along with it’s output (which is the generated token in the desired language) to generate a sentence. This part is called Recurrent Language Model (RLM).</p>
<p>Suggestions for training this model with GPU utilization are welcome.</p>
How to prevent compiling?

sacrophyte: <p>I am working on a laptop and I want to force running on the CPU, and I have set the following to stop TorchDynamo from compiling, but it ignores all my vars:<br>
Get-ChildItem Env: | Where-Object { $_.Name -like “<em>TORCH</em>” }</p>
<p>Name                           Value</p>
<hr>
<p>TORCH_COMPILE_BACKEND          eager<br>
TORCH_COMPILE_SKIP             1<br>
TORCH_COMPILE_WAIT             1<br>
TORCH_DISABLE_DYNAMO           1<br>
TORCHINDUCTOR_CPP_WRAPPER      0</p>
<p>How do I prevent Torch from trying to compile anything, and stop the flood of errors with my python scripts?</p>
Nan output in action

AI_my_dream: <p>I’ve encountered a serious issue: after about 500–1000 iterations, my actions become NaN (e.g., action = [nan, nan, nan, nan, nan, nan]).</p>
<p>Three agents:</p>
<p>TensorDict(</p>
<p>fields={</p>
<pre><code>   action: Tensor(shape=torch.Size(\[6\]), device=cuda:0, dtype=torch.float32, is_shared=True),

   collector: TensorDict(

       fields={

           traj_ids: Tensor(shape=torch.Size(\[\]), device=cuda:0, dtype=torch.int64, is_shared=True)},

       batch_size=torch.Size(\[\]),

       device=cuda:0,

       is_shared=True),

   done: Tensor(shape=torch.Size(\[1\]), device=cuda:0, dtype=torch.bool, is_shared=True),

   loc: Tensor(shape=torch.Size(\[6\]), device=cuda:0, dtype=torch.float32, is_shared=True),

   observation: Tensor(shape=torch.Size(\[30\]), device=cuda:0, dtype=torch.float32, is_shared=True),

   params: TensorDict(

       fields={

           default_x: Tensor(shape=torch.Size(\[3\]), device=cuda:0, dtype=torch.int64, is_shared=True),

           default_y: Tensor(shape=torch.Size(\[3\]), device=cuda:0, dtype=torch.int64, is_shared=True),

           max_movement: Tensor(shape=torch.Size(\[3\]), device=cuda:0, dtype=torch.int64, is_shared=True),

           memory_size: Tensor(shape=torch.Size(\[\]), device=cuda:0, dtype=torch.int64, is_shared=True),

           rand_init: Tensor(shape=torch.Size(\[\]), device=cuda:0, dtype=torch.bool, is_shared=True),

           scale: Tensor(shape=torch.Size(\[\]), device=cuda:0, dtype=torch.int64, is_shared=True),

           user_velocity: Tensor(shape=torch.Size(\[3\]), device=cuda:0, dtype=torch.int64, is_shared=True)},

       batch_size=torch.Size(\[\]),

       device=cuda:0,

       is_shared=True),

   sample_log_prob: Tensor(shape=torch.Size(\[\]), device=cuda:0, dtype=torch.float32, is_shared=True),

   scale: Tensor(shape=torch.Size(\[6\]), device=cuda:0, dtype=torch.float32, is_shared=True),

   step_count: Tensor(shape=torch.Size(\[1\]), device=cuda:0, dtype=torch.int64, is_shared=True),

   step_index: Tensor(shape=torch.Size(\[\]), device=cuda:0, dtype=torch.int64, is_shared=True),

   terminated: Tensor(shape=torch.Size(\[1\]), device=cuda:0, dtype=torch.bool, is_shared=True),

   throughput_mean: Tensor(shape=torch.Size(\[\]), device=cuda:0, dtype=torch.float32, is_shared=True),

   throughput_std: Tensor(shape=torch.Size(\[\]), device=cuda:0, dtype=torch.float32, is_shared=True),

   ue_position: Tensor(shape=torch.Size(\[30, 2\]), device=cuda:0, dtype=torch.float32, is_shared=True)},
</code></pre>
<p>batch_size=torch.Size(<span class="chcklst-box fa fa-square-o fa-fw"></span>),</p>
<p>device=cuda:0,</p>
<p>is_shared=True)</p>
How to do back propagation with loss = ||f_{\Theta + \Delta P}(X) - Y||^2 + ||\Delta P||^2 instead of the usual loss = || f_{\Theta + \Delta}(X) - Y ||^2?

jasonrichdarmawan: <p>Hi,</p>
<p>My goal is to do back propagation to a Transformer Block layer.</p>
<p>My goal: <code>loss = ||f_{\Theta + \Delta P}(X) - Y||^2 + ||\Delta P||^2</code></p>
<p>For context, the usual loss: <code>loss = || f_{\Theta + \Delta}(X) - Y ||^2?</code></p>
<p>The problem is I don’t know how to do <code>||f_{\Theta + \Delta}(X) - Y||^2</code> or regularize the <code>||\Delta P||^2</code></p>
<p>What I did:</p>
<ol>
<li>After <code>loss.backward()</code>, I do <code>param.grad = torch.matmul(param.grad, P)</code>. However, I believe this is not correct. For example, even though after <code>loss.backward()</code> the <code>param.grad</code> is accessible, how I can compute the regularization loss <code>||\Delta P||^2</code>?</li>
</ol>

Arunprakash-A: <p>It looks like you want to add the L2 regularisation. There are many ways of achieving it. Assuming that you are using a simple <a href="https://docs.pytorch.org/docs/stable/generated/torch.optim.SGD.html" rel="noopener nofollow ugc">SGD</a> optimizer, I recommend you set <code>weight_decay</code> to some non-zero value while initializing the optimizer. For example,<br>
<code>optimizer = optim.SGD(model.parameters(), lr=0.01, weight_decay=1e-4)</code></p>
<p>This will automatically apply  L2 penalty while updating the parameters. Of course, you can use any optimizer of your choice from pytorch.</p>

jasonrichdarmawan: <p>I want the neural network to update only along the subspace defined by P</p>
<p>〖〖Θ ̃=(arg min)_Δ ̂ 〗⁡‖f_(Θ+Δ ̂ ) (X)-Y‖〗^2<br>
〖Θ ̃=(arg min)┬Δ ̂ 〗⁡〖‖f_(Θ+Δ ̂P) (X)-Y‖^2+‖Δ ̂P‖^2 〗</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/7/d796002bf4223422b41c545558d3530b912c094e.png" data-download-href="https://discuss.pytorch.org/uploads/default/d796002bf4223422b41c545558d3530b912c094e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d796002bf4223422b41c545558d3530b912c094e_2_690x431.png" alt="image" data-base62-sha1="uLa1VyjqILXEsvSYTeH036EmDAG" width="690" height="431" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d796002bf4223422b41c545558d3530b912c094e_2_690x431.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d796002bf4223422b41c545558d3530b912c094e_2_1035x646.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/7/d796002bf4223422b41c545558d3530b912c094e_2_1380x862.png 2x" data-dominant-color="E8E8E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×1200 72.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Is it normal for compiled model outputs to have different tensor strides?

tinywisdom: <p>Hello everyone,</p>
<p>I recently compiled my model using <code>torch.compile</code>, and I noticed that the results after compilation have some layout differences compared to the original (eager) execution.</p>
<p>When I check the outputs using <code>torch.testing.assert_close</code>, I sometimes get messages like:</p>
<pre><code class="lang-auto">The values for attribute 'stride()' do not match: (27, 1, 9, 3) != (27, 9, 3, 1)
</code></pre>
<p>or</p>
<pre><code class="lang-auto">The values for attribute 'stride()' do not match: (1, 2) != (2, 1)
</code></pre>
<p>The numerical values themselves appear identical, but the stride orders differ.<br>
I’d like to ask: <strong>is this expected or normal behavior after model compilation?</strong><br>
Does the compiler sometimes change the memory layout (e.g., due to optimization or contiguous conversion)?</p>
<p>Thanks a lot for your time and for maintaining this great project!</p>

ptrblck: <p>Yes, this is expected. From the <a href="https://docs.google.com/document/d/1y5CRfMLdwEoF1nTk9q8qEu1mgMUuUtvhklPKJ2emLU8/edit?tab=t.0">missing manual</a>:</p>
<blockquote>
<p>Stride divergence<br>
PT2 is not guaranteed to produce tensors with exactly the same stride as the eager variants. This can potentially cause correctness problems (or runtime crashes) outside of the torch.compile region if subsequent code is expecting a specific striding without testing, or is using the limited subset of APIs (mostly torch.as_strided, and also obscure situations involving mutating through .reshape()/.contiguous()) which are sensitive to input of strides.</p>
<p>TODO: There is supposed to be a way to fix strides, similar to how we can do this for custom ops.</p>
</blockquote>

GerryMan: <p>That’s normal. torch.compile can change the tensor layout sometimes for speed, the values themselves stay fine</p>

tinywisdom: <p>Thanks a lot for the clarification! That makes sense — good to know it’s expected behavior with <code>torch.compile</code>. Appreciate the detailed explanation!</p>
Which multi-GPU setups are worthwhile for training VLM's?

GabrielFerrante: <p>Hello everyone,</p>
<p>I have a project to train a VLM model. I would like to know if it is worth having two GPUs or one high-end GPU.</p>
<p>I saw some setups with two RTX 3060 8GBs and was curious if that would provide enough performance for training.<br>
But looking at some GPUs like the 4080, the number of CUDA cores is almost double.</p>
<p>Another question I have is whether Pytorch allows multi-GPU training with Nvidia GPUs from different generations. For example, RTX 3060 + RTX 4070, both with differences in VRAM size.</p>

GerryMan: <p>Two 3060s can work but one 4080’s faster and less hassle. Mixed gpus work but don’t perform great together…</p>
Adding augmentation to both image and segmentation mask

crossentropy: <p>I am lost, i have issues augmenting both image and masks correctly in the sense that at times my image is flipped and the mask is not or the mask is flipped and the image is not. Check the code out below:</p>
<pre><code class="lang-auto">im_size = (256,256)
device = "cuda" if torch.cuda.is_available() else "cpu"

train_transform = v2.Compose(
    [
        v2.Resize(im_size),
        v2.RandomHorizontalFlip(p = 0.5),
        v2.RandomVerticalFlip(p = 0.5),
        # v2.RandomPerspective(distortion_scale = .16, p = 1.0),
        # v2.RandomRotation(degrees = [90, 180]),
        v2.ToImage(),
        v2.ToDtype(dtype = torch.float32, scale = True),
    ]
)

val_transform = v2.Compose(
    [
        # v2.Resize(im_size),
        v2.ToImage(),
        v2.ToDtype(dtype = torch.float32, scale = True),
    ]
)


class barbadosLandSurveyDataset(Dataset):
    def __init__(self, df_:pd.DataFrame, tf):
        self.df_ = df_
        # self.transform = transform
        self.tf = tf

    def __len__(self):
        return len(self.df_)

    def __getitem__(self, idx):
        im_ = Image.open(self.df_.iloc[idx, :]["image_path"]).resize((256,256))
        w,h = im_.size
        mask_ = Image.fromarray((polygon_to_mask(polygon = self.df_.iloc[idx, :]["geometry"])*255.0)).convert("L")
        im_, mask_ = self.tf((im_, mask_))
        # im_ = v2.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])(im_)
        return im_, mask_
        


def return_dataloader(ds, bsz:int, shuffle:bool, num_workers:int, pin_memory:bool, persistent_workers:bool):
    dl = DataLoader(dataset = ds,
                    batch_size = bsz,
                    shuffle = shuffle,
                    num_workers = num_workers,
                    pin_memory = pin_memory,
                    persistent_workers = persistent_workers)
    return dl
</code></pre>
<p>what could possibly be wrong?</p>

ptrblck: <p>Could you try to use <code>tv_tensors</code> as described <a href="https://discuss.pytorch.org/t/torchvision-v2-transforms-not-applying-to-both-image-and-mask/222034/3">here</a>?</p>
<p>Alternatively, you could also use the functional API as given in <a href="https://discuss.pytorch.org/t/torchvision-transfors-how-to-perform-identical-transform-on-both-image-and-target/10606/7">this example</a>.</p>
FSDP2 for GGUF / BNB models

komikndr: <p>I know this question sounds a bit crazy, but I need to shard a GGUF or BitsAndBytes model, or any dtype lower than FP8.</p>
<p>The problem is that when I try to load a distributed <code>state_dict</code>, the model either fails to load or the forward pass breaks due to mismatched GGML tensor shapes during matrix multiplication. For example, if I load a model like <strong>Wan 2.1 1.3B</strong>, its native dtype in one of its attention layer is <strong>1536×1536</strong>, but when quantized to GGUF and loaded into non FSDP models, it becomes <strong>1536×864</strong>.</p>
<p>This works fine for <strong>single-GPU inference</strong>, but not with <strong>FSDP2</strong> when using distributed <code>state_dict</code>. I often get errors like <em>“cannot copy meta tensor”</em> due to shape mismatches between the model and the quantized weights.</p>
<p>Has anyone tried something similar? Or is this just a waste of time to pursue?</p>
<p>For reference:</p>
<ul>
<li><strong>PyTorch 2.8</strong>, <strong>CUDA 12.8</strong>, <strong>Python 3.11</strong>, running on <strong>WSL</strong> with an <strong>RTX 3090</strong>.</li>
</ul>
<p>Thanks for your time</p>
Update gradients based on worst loss in set

dicaes: <p>Hi everyone! I’m still a bit unfamiliar with pytorch so my apologies if this is a dumb question.</p>
<p>I have some data where each datapoint is a set of 10 vectors measuring the state of the same object in different circumstances. In total, the dataset comprises about 20k such sets each relating to a similar (but distinct) object and a label identifying each object as one of 3 classes.</p>
<p>I want to train a simple model to classify of these objects robustly and, in order to do this, I want to compute the cross entropy loss for each vector and update my gradients based on the worst one. I tried doing the following but I think it isn’t computing the gradients before the max() as it isn’t learning at all.</p>
<pre data-code-wrap="python"><code class="lang-python">class WorstLoss(nn.Module):
    def __init__(self):
        super().__init__()
        self.cross_entropy = nn.CrossEntropyLoss(reduction='none')

    def criterion(self, inputs, target):
        loss = self.cross_entropy(inputs, torch.ones(10).long().cuda() * target)
        return loss.max()

    def forward(self, inputs, target):
        batched_criterion = torch.vmap(self.criterion)
        return batched_criterion(inputs, target)
</code></pre>
<p>Is there a better approach to this? I might be missing something obvious, my apologies if that is the case. Any help would be much appreciated. Thank you for your time!</p>

valerian.rey: <p>Hi! Just to be sure that I understand everything (please correct me if I’m wrong):</p>
<ul>
<li>Parameter <code>inputs</code> of criterion is of shape <code>(10,)</code>. All of them are obtained independently from a different input vector.</li>
<li>Parameter <code>target</code> of criterion is a scalar, so <code>torch.ones(10).long().cuda() * target</code> is of shape <code>(10,)</code>.</li>
<li>Parameter <code>inputs</code> of forward is of shape <code>(batch_size, 10)</code>.</li>
<li>Parameter <code>target</code> of forward is of shape <code>(batch_size,)</code>.</li>
</ul>
<p>I don’t see anything that seems obviously wrong here. As a sanity check, have you tried replacing <code>loss.max()</code> by <code>loss.mean()</code>? Maybe it’s actually the idea of minimizing the worst loss that prevents your model from training. Say that among your 10 vectors, one is always pure noise. It will likely have the worst loss among the 10. Using the worst loss prevents you entirely from training based on any of the other 9 vectors, so you wouldn’t learn anything in this scenario.</p>

dicaes: <p>Hi! Thank you for your response! Your analysis was spot on! I did indeed end up using a weighed mean which did perform much better. Thank you for your help!</p>

valerian.rey: <p>No worries!</p>
<p><em>Self-promotion: You could check my multi-objective optimization library <a href="https://github.com/TorchJD/torchjd" rel="noopener nofollow ugc">TorchJD</a> if you wanna try other ideas than just taking the mean loss.</em></p>
AttributeError: 'StableDiffusionPipelineOutput' object has no attribute 'frames'

Casey_KimScripts: <p>Like in Title this AttributeError pops up whenever I ran this Huggingface code</p>
<pre><code class="lang-auto">from diffusers import StableDiffusionPipeline
</code></pre>
<pre><code class="lang-auto">import sys
</code></pre>
<pre><code class="lang-auto">import torch
</code></pre>
<pre><code class="lang-auto">import cv2
</code></pre>
<pre><code class="lang-auto">import numpy as np
</code></pre>
<pre><code class="lang-auto">sys.path.append(r"/Users/My/Path/ToProj")
</code></pre>
<pre><code class="lang-auto">pipe = StableDiffusionPipeline.from_pretrained("runwayml/stable-diffusion-v1-5")
</code></pre>
<pre><code class="lang-auto">pipe = pipe.to("cpu")
</code></pre>
<pre><code class="lang-auto">prompt = "A flying Pusheen in the early morning with matching flying capes. The Pusheen keeps flying. The Pusheen keeps flying with some Halloween designs."
</code></pre>
<pre><code class="lang-auto">negative_prompt = "Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality, low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards"
</code></pre>
<pre><code class="lang-auto">
</code></pre>
<pre><code class="lang-auto">output = pipe(
</code></pre>
<pre><code class="lang-auto">    prompt=prompt,
</code></pre>
<pre><code class="lang-auto">negative_prompt=negative_prompt,
</code></pre>
<pre><code class="lang-auto">height=480,
</code></pre>
<pre><code class="lang-auto">width=832,
</code></pre>
<pre><code class="lang-auto">num_frames=81,
</code></pre>
<pre><code class="lang-auto">guidance_scale=5.0
</code></pre>
<pre><code class="lang-auto">).frames[0]
</code></pre>
<pre><code class="lang-auto">export_to_video(output, "outputPusheen.mp4", fps=15)
</code></pre>
<p>Do I have to define frames somewhere or does this error refer to another cause?</p>
No module named cv2 although opencv has been installed already

Casey_KimScripts: <p>I wanted to try the simple code which was supposed to convert text to video like this</p>
<pre><code class="lang-auto">import cv2
import numpy as np

cap = cv2.VideoCapture(0) 
#more lines of code follow
</code></pre>
<p>I tried moving cv2.so around in my source code folder and everywhere. I tried even uninstalling and re-installing opencv then running this code over and over again. My terminal on VSCode gets terminated by error <code>No module named cv2</code> - none of my efforts helped. I am about to bang off my keyboard out of frustration.</p>
<p>Could anyone kindly tell me what went wrong here?</p>

ptrblck: <p>What does <code>pip list | grep opencv</code> return?</p>
<p>If it’s empty, run <code>pip install opencv-python</code>.</p>

Casey_KimScripts: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="223297">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>pip list | grep opencv</p>
</blockquote>
</aside>
<p>This command line gave me this output <code>opencv-python      4.12.0.88</code><br>
I just tried running a python code with cv2 but still got <code>no module named cv2.</code>.  No idea what went wrong here</p>

Arunprakash-A: <p>Try <code>import sys</code> and <code>print(sys.path)</code>. See whether the path to OpenCV is listed in the output. Carefully check the list of paths.</p>

jeffrey-ke: <p>After putting <code>cv2.so</code> back, I’d also check if you’re using the right python executable to run your script. Are you using a miniconda environment? That’s always the recommended way of managing dependencies like opencv and torch.</p>

twoudsma: <p>Just to give an alternative to miniconda, you can also try using <code>uv</code> as a package manager and run the script from inside a virtual environment, or directly with the <code>uv</code> command.</p>

Casey_KimScripts: <p>Thank you so much everyone for the tips - I just figured out myself with your assistance.</p>
<p>Following Arunprakash’s tips, I noticed that openCV has not been installed while running print(sys.path).</p>
<p>This is because my Dockerfile does not have it installed (for deploying cv2 with Docker please see this thread <a href="https://discuss.pytorch.org/t/building-pytorch-2-6-0-from-source-on-macos-x86-64-intel/216283" class="inline-onebox">Building PyTorch 2.6.0 from source on macOS x86_64 (Intel)</a> )</p>
<p>So I had to add this line into my Dockerfile</p>
<p><code>RUN pip install opencv-python-headless</code></p>
<p>And also I had to add this line into the source code</p>
<p><code>sys.path.append(r"/Users/Path/To/MyPytorchProject")</code></p>
<p>Then the error is gone.</p>
Workflow for lowerimg a checkpoint from civitai

jdumke: <p>Hello all,</p>
<p>I’m going fully bananas by reading tons of documentation [1, 2] and many more.</p>
<p>The point of struggle is extracting the state_dict from the safetensors file with the checkpoint.</p>
<p>I can load the data from the file in many variants, but none of these leads a step further and let me extract the desired parts.</p>
<p>Could somebody give me a description of a typical workflow?</p>
<p>Greets,</p>
<p>Jonathan</p>
<hr>
<p>[1] <a href="https://huggingface.co/docs/transformers/executorch" rel="noopener nofollow ugc">Tranformers Executorch preexporter</a></p>
<p>[2] <a href="https://docs.pytorch.org/executorch/stable/index.html" rel="noopener nofollow ugc">Executorch docs</a></p>
Is there a PyTorch build that supports NVIDIA RTX 5090 (Compute Capability 12.0 / sm_120)?

David_Ariza: <p>Hey everyone,</p>
<p>I just got an <strong>RTX 5090</strong>, and I’m running into issues using it with PyTorch.<br>
It looks like PyTorch doesn’t yet support the new <strong>Blackwell architecture</strong> (compute capability 12.0 / sm_120).</p>
<p>When I try to run any CUDA ops, I get this warning and crash:</p>
<p><code>UserWarning: NVIDIA GeForce RTX 5090 with CUDA capability sm_120 is not compatible with the current PyTorch installation.</code><br>
<code>The current PyTorch install supports CUDA capabilities sm_61 sm_70 sm_75 sm_80 sm_86 sm_90.</code><br>
<code>torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device</code></p>
<p>I’m currently on the latest stable build (CUDA 12.6).<br>
Before switching to the nightly, I wanted to ask:</p>
<ul>
<li>
<p>Is there <em>any</em> PyTorch build (nightly or experimental) that already supports <strong>sm_120</strong> or any plan for it?</p>
</li>
<li>
<p>Or should I wait for a future release (like PyTorch 2.5 or the CUDA 13 build)?</p>
</li>
<li>
<p>If it’s not out yet, can I build from source with CUDA 13 to get it working?</p>
</li>
</ul>
<p>Would love to hear if anyone has managed to get their RTX 5090 / 5080 working yet.</p>

ptrblck: <p>Blackwell GPUs require CUDA 12.8+ so install any stable or nightly PyTorch binary with CUDA 12.8+ from our install matrix.</p>
The right way to use FSDP via accelerate

Jithendaraa_Subraman: <p>I am coming across some behavior I am unable to explain:</p>
<ol>
<li>From the docs, my understanding is that setting <code>batch_size=64</code> in my dataloaders with <code>split_batches=False</code> for accelerate would have an effective batch size of 512. This runs without issues. However, <code>batch_size=512</code> with <code>split_batches=True</code> would also have the same global batch size but this gives me OOM. Shouldn’t both these approaches give me the same results (or at least no OOMs)?</li>
<li>I also would like to know the right way to call non-forward methods of my <code>model</code> .<br>
Is the recommended way to always run it within an <code>FSDP.summon_all_gather()</code>? In what cases should I set <code>ignored_modules</code> in the FullyShardedDataParallelPlugin*,* and when should I choose otherwise?</li>
</ol>
<p>Here’s a tiny code snippet showing how I intend to use <code>model.generate()</code></p>
<pre><code class="lang-auto">for iter, batch in enumerate(train_loader):
    outputs = model(batch)
    accelerator.backward(outputs.loss)
    optimizer.step()
    scheduler.step()

    if iter % 5 == 0:
        with torch.inference_mode(), FSDP.summon_full_params(model):
            eval_text = model.generate(batch)
</code></pre>
Getting different accuracy while keeping everything same

Rewant_Angira: <p>I am making a simple ANN model in pytorch i am using SGD optimizer with learining rate 3e-4 this got me an test accuracy around 79 to 80 % but when i changed it to Adam optimizer and again to SGD and ran my code it gave me test accuracy of 66% why is that and how can i fix it?</p>

jwilk: <p>Hi, do I understand correctly that you trained your network first using SGD and then trained the second one using first Adam optimizer and then fine tuned using SGD optimizer? Or you trained both networks using just SGD optimizers?</p>
<p>Also, do you set <code>torch.manual_seed()</code>? As far as I know it is also necessary to set <code>torch.backends.cudnn.deterministic = True</code> and <code>torch.backends.cudnn.benchmark = False</code> for full reproducibility.</p>