akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why does autograd.backward go one edge further than `inputs`?

username: Aaron_Kriegman
cooked: <p>Consider the following example (REPL pasteable)</p>
<pre><code class="lang-auto">import torch

class LoggingFn(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        ctx.save_for_backward(x)
        return x * 2
    @staticmethod
    def backward(ctx, grad_output):
        print("Backward called on LoggingFn")
        return grad_output * 2

a = torch.tensor(1.0, requires_grad=True)
b = LoggingFn.apply(a)
c = LoggingFn.apply(b)
loss = LoggingFn.apply(c)

loss.backward(inputs=[c])

print("a.grad =", a.grad)
print("b.grad =", b.grad)
print("c.grad =", c.grad)
</code></pre>
<p>prints</p>
<pre><code class="lang-auto">&gt;&gt;&gt; loss.backward(inputs=[c])
Backward called on LoggingFn
Backward called on LoggingFn
&gt;&gt;&gt; print("a.grad =", a.grad)
a.grad = None
&gt;&gt;&gt; print("b.grad =", b.grad)
b.grad = None
&gt;&gt;&gt; print("c.grad =", c.grad)
c.grad = tensor(2.)
</code></pre>
<p><code>LoggingFn.backward</code> is called twice, meaning it backpropogates back to <code>b</code>. It was correct to not go all the way back to <code>a</code>, but why didn’t it stop at <code>c</code>? Is this incorrect behavior, or is there a reason to go one edge further than needed?</p>

username: Aaron_Kriegman
cooked: <p>Here’s another weird example:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; a = torch.tensor(1.0, requires_grad=True)
&gt;&gt;&gt; b = LoggingFn.apply(a)
&gt;&gt;&gt; c = LoggingFn.apply(b)
&gt;&gt;&gt; loss = LoggingFn.apply(c)
&gt;&gt;&gt; loss.backward(inputs=[c])
Backward called on LoggingFn
Backward called on LoggingFn
&gt;&gt;&gt; loss.backward(inputs=[c])
Backward called on LoggingFn
Backward called on LoggingFn
&gt;&gt;&gt; loss.backward(inputs=[c])
Backward called on LoggingFn
Backward called on LoggingFn
&gt;&gt;&gt; c.grad
tensor(6.)
&gt;&gt;&gt; c.backward(gradient=c.grad)
Backward called on LoggingFn
Backward called on LoggingFn
&gt;&gt;&gt; a.grad
tensor(24.)
&gt;&gt;&gt; c.grad
tensor(12.)
</code></pre>
<p>We backprop from <code>loss</code> to <code>c</code> three times, and the gradients are correctly accumulated and backproped to <code>a</code>. But for some reason this also changed the gradient saved in <code>c</code>.</p>
<p>I guess this one makes sense. The argument <code>gradient=c.grad</code> was accumulated into the already saved <code>c.grad</code>.</p>

username: soulitzer
cooked: <p>Indeed we shouldn’t compute the extra backprop. It’s just the current state due to simplicity of implementation (see the note in the doc). Feel free to file an issue if you have a perf-sensitive use case and we can probably try to fix it.</p>

username: valerian.rey
cooked: <p>Hi! As an additional note, this is only a problem with the <code>backward</code> function. If you care about performance and calling one extra node is too costly, you can also call <code>torch.autograd.grad</code> instead of <code>torch.autograd.backward</code> or <code>tensor.backward</code>, and the extra node will not be called.</p>
<p>The <code>.grad</code> fields will have to be accumulated manually, though.</p>
PyTorch equivalent of exclusive cumsum?

username: AlphaBetaGamma96
cooked: <p>Hi All,</p>
<p>I was wondering if there’s a way to do an exclusive cumsum (like that implemented with Tensorflow’s cumsum). An example of this would be,</p>
<pre><code class="lang-auto">tf.cumsum([a,b,c], exclusive=False) =&gt; [a, a+b, a+b+c]  #standard cumsum
tf.cumsum([a, b, c], exclusive=True)  =&gt; [0, a, a + b]  #exclusive cumsum
</code></pre>
<p>Let’s say I have a Tensor of [1,2,3] and I take the cumsum of it. That would return a Tensor of [1,3,6]. However, if I wanted to take the “exclusive” cumsum which would be [0,1,3]? How exactly could be done efficiently? Are there plans to add this feature to PyTorch?</p>
<p>Thanks in advance! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: KFrank
cooked: <p>Hi Alpha!</p>
<aside class="quote no-group quote-modified" data-username="AlphaBetaGamma96" data-post="1" data-topic="107259" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3da27b/48.png" class="avatar"> AlphaBetaGamma96:</div>
<blockquote>
<pre><code class="lang-auto">tf.cumsum([a, b, c], exclusive=True)  =&gt; [0, a, a + b]  #exclusive cumsum
</code></pre>
<p>Let’s say I have a Tensor of [1,2,3]. However, if I wanted to take the “exclusive” cumsum which would be [0,1,3]?</p>
</blockquote>
</aside>
<p>The best I can think of is to use pytorch’s “standard” <code>cumsum()</code> and<br>
use <code>roll()</code> to right-shift the result:</p>
<pre data-code-wrap="python"><code class="lang-python">&gt;&gt;&gt; t = torch.tensor ([1, 2, 3])
&gt;&gt;&gt; res = t.cumsum (0).roll (1, 0)
&gt;&gt;&gt; res[0] = 0
&gt;&gt;&gt; print (res)
tensor([0, 1, 3])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: AlphaBetaGamma96
cooked: <p>Hi K. Frank!</p>
<p>That solution looks fantastic!</p>
<p>Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: Max_Wu
cooked: <p>Hi, exclusive = torch.cumsum(a) - a</p>

username: vwxyzjn
cooked: <p>Very nice. What an elegant solution! I register an account just to upvote your solution <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: mrTsjolder
cooked: <p>Note that this elegant solution might result in <a href="https://en.wikipedia.org/wiki/Catastrophic_cancellation" rel="noopener nofollow ugc">catastrophic cancellation</a>, especially when moving to low precision dtypes.</p>
FSDP1/FSDP2 sharding across CPUs?

username: hpcpony
cooked: <p>Does FSDP1 and/or FSDP2 work on GPU only?  I didn’t see any explicit documentation that said so.  Conceptually I don’t see why there much difference sharding across GPUs (processes) vs. CPUs (prosesses) but it doesn’t look like the Pytorch FSDP can’t do it.    I was looking to shard across CPU processes bound to NUMA nodes within a single machine.</p>

username: weifengpy
cooked: <p>FSDP1/FSDP2 use cuda wait_event and wait_stream heavily to achieve compute communication overlapping. that’s why we always assume cuda/rocm availability</p>
<p>there is ongoing effort to make FSDP2 + Gloo work: <a href="https://github.com/pytorch/pytorch/pull/156883" class="inline-onebox" rel="noopener nofollow ugc">Work: wait_stream API by d4l3k · Pull Request #156883 · pytorch/pytorch · GitHub</a><br>
cc <a class="mention" href="/u/d4l3k">@d4l3k</a> if you have more info</p>

username: hpcpony
cooked: <p>Fair enough.  I was afraid I was doing something wrong.</p>
Efficient Computation of Second-Order Gradients

username: katonapng
cooked: <p>Hi everyone,</p>
<p>I’m working on computing the score matching objective for a softcore potential, and I’m running into performance issues when calculating the second-order gradients (i.e., the divergence of the gradient field).</p>
<p>Here’s a simplified example of what I’m doing:</p>
<pre data-code-wrap="python"><code class="lang-python">def minimal_test(x):
    xi = x[:, 0, :]
    xj = x[:, 1, :]
    diff = xi - xj
    r = torch.sqrt(torch.sum(diff ** 2, dim=-1) + 1e-10)
    sigma = torch.exp(torch.tensor(0.15))
    k = torch.sigmoid(torch.tensor(0.5))
    phi = torch.pow(sigma / r, 2 / k)
    
    energy = phi.sum(dim=-1)
    return -energy

x_t = torch.tensor([[[0.4806, -0.5267], [0.5513, 0.6484]]], requires_grad=True)  # Shape: (B, N, D)

def psi(x_t):
    output = minimal_test(x_t)
    gradients = torch.autograd.grad(
        outputs=output, inputs=x_t, grad_outputs=torch.ones_like(output),
        retain_graph=True, create_graph=True,
    )[0]
    return gradients

gradients = psi(x_t)

# Compute divergence via second-order derivatives (inefficient!)
divergence = torch.zeros(x_t.shape[0], x_t.shape[1])
for d in range(x_t.shape[-1]):
    for i in range(x_t.shape[-2]):
        second_grad = torch.autograd.grad(
            gradients[:, i, d].sum(), x_t, create_graph=True)[0][:, i, d]  # shape: (B,)
        divergence[:, i] += second_grad

print("Second Gradients (autograd per input):", divergence)
</code></pre>
<p>The issue is that this per-element loop over <code>i</code> and <code>d</code> is very slow, especially when scaling up to more particles or higher dimensions. I tried using <code>vmap</code>, but couldn’t make it work due to the indexing <code>gradients[:, i, d]</code>, which seems to break batching.</p>
<p>Is there any cleaner or more efficient way to compute this divergence without having to loop over each element individually?</p>
<p>Thanks in advance for any help or pointers!</p>

username: paulk
cooked: <p>If I understand correctly you are computing the trace of the hessian (aka the laplacian)? There is a function for computing the <a href="https://docs.pytorch.org/docs/stable/generated/torch.autograd.functional.hessian.html" rel="noopener nofollow ugc">hessian</a>.<br>
Since you only need the elements on the diagonal this might compute a lot of elements you don’t need but I still think there might be a chance that this is faster?<br>
Efficient implementations (using vmap to handle vectorization) following that idea have also been discussed <a href="https://discuss.pytorch.org/t/implementing-calculation-of-the-laplacian/99644/5">here</a>.</p>
<p>If this is still too inefficient (and this might be a moonshot) there are <a href="https://www.sciencedirect.com/science/article/pii/S0045782524001397#:~:text=Hutchinson%20Trace%20Estimation%20(HTE)%20%5B12%5D%20is%20an%20unbiased,has%20been%20made%20for%20improvement." rel="noopener nofollow ugc">methods for estimating the trace of the hessian</a> but they are more involved.</p>
<p>hope that helps</p>
Small loss in training, large loss after loading .pkl

username: paul188
cooked: <p>I am training a PointNet on pointclouds with multiple channels.<br>
During training, I get a minimal MSE training loss of about ~0.0003.<br>
And a validation loss of ~0.003.<br>
Unfortunately, when I save my network using the code</p>
<pre><code class="lang-auto">if (snapshot_ticks is not None) and (done or cur_tick % snapshot_ticks == 0):
            data = dict(ema=ema, loss_fn=loss_fn, augment_pipe=augment_pipe, dataset_kwargs=dict(dataset_kwargs))
            for key, value in data.items():
                if isinstance(value, torch.nn.Module):
                    value = copy.deepcopy(value).eval().requires_grad_(False)
                    misc.check_ddp_consistency(value)
                    data[key] = value.cpu()
                del value # conserve memory
            if dist.get_rank() == 0:
                with open(os.path.join(run_dir, f'network-snapshot-{cur_nimg//1000:06d}.pkl'), 'wb') as f:
                    pickle.dump(data, f)
            del data # conserve memory
</code></pre>
<p>And reload it for inference later, using</p>
<pre><code class="lang-auto"> f = open(network_pkl, 'rb')
 net = pickle.load(f)['ema'].to(device)
 net.eval()
</code></pre>
<p>I get a loss of ~ 0.1 - 0.2. My loss is computed using a MSE:</p>
<pre><code class="lang-auto">class MSELoss(torch.nn.Module):
    """
    Mean Squared Error Loss for regression tasks.
    """
    def __init__(self):
        super(MSELoss, self).__init__()
        self.mse_loss = torch.nn.MSELoss()

    def __call__(self, net, data, augment_pipe=None):
        """
        Compute the MSE loss between predictions and targets.
        
        Args:
            pred (torch.Tensor): Predicted values.
            target (torch.Tensor): Ground truth values.
        
        Returns:
            torch.Tensor: Computed MSE loss.
        """
        input = data[:, :-1, :]      # shape: [600, N-1, 50]
        target = data[:, -1, :]      # shape: [600, 50]
        pred = net(input)            # shape: [600, 1, 50]

        pred = pred.squeeze(1)       # shape: [600, 50]

        return self.mse_loss(pred, target)
</code></pre>
<p>Thanks in advance for any help ! I can’t seem to figure this out.</p>

username: ptrblck
cooked: <p>Try to narrow down which part of your pipeline is causing the difference in the outputs. I.e. you could compare the outputs using static inputs (e.g. using <code>torch.ones</code>) before and after loading the model. If the model outputs differ restoring the model is most likely broken and you should check if all parameters and buffers were loaded properly.<br>
If this test works fine, compare the data loading and processing pipeline as the difference could be introduced in this part of the pipeline.</p>
CUDA support for RTX 50x on Windows

username: ntp7
cooked: <p>Hi,</p>
<p>Any word when there will be full pytorch support for sm_120 on Windows? AFAIU it’s currently only on Linux? I’ve just upgraded GPU and I’m now getting a bunch of errors like this -</p>
<p>G:\framepack\framepack_cu126_torch26\system\python\lib\site-packages\torch\cuda_<em>init</em>_.py:235: UserWarning:<br>
NVIDIA GeForce RTX 5060 Ti with CUDA capability sm_120 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_61 sm_70 sm_75 sm_80 sm_86 sm_90.<br>
If you want to use the NVIDIA GeForce RTX 5060 Ti GPU with PyTorch, please check the instructions at <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">https://pytorch.org/get-started/locally/</a></p>
<p>The nightlies don’t work, because I’ve tried that. Very frustrating having a racehorse and having to ride a donkey. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>PyTorch supports all Blackwell architectures on Linux and Windows since its 2.7.0 release (and beforehand via nightlies) via its CUDA 12.8 builds.</p>

username: ntp7
cooked: <p>Hi, thanks very much for your response. Unfortunately despite your claim, I’m still getting errors on my older pytorch apps like the Framepack one I included in my original post.</p>
<blockquote>
<blockquote>
<p>NVIDIA GeForce RTX 5060 Ti with CUDA capability sm_120 is not compatible with the current PyTorch installation.</p>
</blockquote>
</blockquote>
<p>Is there a disconnect somewhere?</p>
<p>Thanks again.</p>

username: ptrblck
cooked: <p>Yes, based on the error message you are installing a PyTorch binary using older CUDA runtime dependencies and need to uninstall it and install any PyTorch binary with CUDA 12.8 by selecting it from our <a href="https://pytorch.org/get-started/locally/">install matrix</a>.</p>
Named arguments and dict output with AOTInductor

username: rdgq
cooked: <pre data-code-wrap="cpp"><code class="lang-cpp">  std::vector&lt;at::Tensor&gt; AOTModelPackageLoader::run(
      const std::vector&lt;at::Tensor&gt;&amp; inputs,
      void* stream_handle = nullptr);
</code></pre>
<p>I’d like to use AOTInductor to do inference for a PyTorch model whose forward method returns a <code>dict[str, Tensor]</code>. The forward method takes a large number of arguments as well, so ideally for robustness I’d be able to call it with named arguments (ie. a <code>GenericDict</code>) from the C++ side. However based on the above signature, you must only run the model with a vector of tensors as input and expect only a vector of tensors as output. Is there another way to run the module but have it take and expect arbitrary data structures like an arbitrary <code>IValue</code>?</p>
Feeding categorical information into a GAN discriminator

username: 40uf411
cooked: <p>Hi,</p>
<p>I am running a set up where the generator is 3D and the discriminator is 2D.</p>
<p>Feeding the discriminator random slices from all three axis does not work, because the discriminator can then not distinguish between the differences in structure between the three planes.</p>
<p>I wanted to ask you whats the SOTA way of incorporating this information into the discriminator.<br>
Also, should I feed this information to the input layer of the model or to every convolutional block/level.</p>
<p>Thanks in advance.</p>

username: 40uf411
cooked: <p>If I am not mistaken, this technic was from the cGAN paper, but, there are newer advancements since then I would assume. maybe not directly from gan papers, but anything would help.</p>
Graph reconstruction using autoencoder

username: ag1805x
cooked: <p>I am trying to build a GNN model, using PyG, to generate embeddings for a list of graphs to be used for a further downstream task like unsupervised clustering. The overall task is similar to using methods like graph2vec to generate whole graph embeddings. However, in my case, I also have node attributes and optionally edge weight and/or attributes. My idea was to train a model that could take into consideration the graph structure, the node attributes and if available the edge weights/attributes to generate the graph embeddings. Since I do not have any labels for the graphs, I decided to use an autoencoder approach to guide the training process. I observed that the decoder is unable to correctly reconstruct the graph structure particularly.</p>
<p>Networks:: Directed networks with ~1500 nodes and an average density of ~0.05. All the networks contain the same nodes but the edges differ. The nodes contain three attributes (one positive continuous and two binary). The assortativity coefficient is close to 0 (~0-0.2).</p>
<p>Encoder:: Intakes the edge indices and node attributes. Message passing occurs through three layers of GATConv before using AttentionalAggregation to convert the node level embeddings to graph level-embeddings. After each convolution layers ELU activation is used followed by dropout.</p>
<p>Decoder:: For decoding the structure, the graph-level embeddings are first decoded to node level embeddings using a simple two-layered feedforward NN with ReLU activation followed by using another similar  feedforward NN to obtain the edge logits for all possible edges (adjacency matrix). For decoding the node attributes, I directly pass the graph-level embeddings through a similar two-layered feedforward NN. I have previously tried the InnerProductDecoder for decoding the structure but that dis not work well.</p>
<p>Losses: For structure, calculating the BCELossWithLogits on the whole adjacency matrix (used pos_weight to account for the sparse network). For the node attributes using the MSE loss. The sum is sent backward.</p>
<p>With the above setting, I observed that I reached an AUC of ~0.65 on the adjacency matrix reconstruction.  As a cross-check I also calculated the MSE loss on the node embeddings from the encoder and the decoder which seems to be low.</p>
<p>Questions:<br>
(1) How can I improve the graph reconstruction when my input network has a very less density?<br>
(2) Do I need to be cautious about heterophily and use a heterophily aware model?<br>
(3) How relevant should the node attributes be in predicting the edge? Is it necessary for GNN to have node attributes that are capable of edge prediction?</p>
<p>I am new into the domain of neural networks. Any help will be appreciated.</p>
<p>[Note: Replicate query on <a href="https://stackoverflow.com/questions/79692392/graph-reconstruction-using-autoencoder" rel="noopener nofollow ugc">Stack Overflow</a>]</p>
Cannot create a mask for each sequence in a batch with Flex Attention

username: Luis_Lopes
cooked: <p>Hello everyone,</p>
<p>I have been experimenting with Flex Attention when I ran into a problem that makes Flex Attention useless for my use case:</p>
<pre data-code-wrap="python"><code class="lang-python">def causal_mask(b, h, q_idx, kv_idx):
    return q_idx &gt;= kv_idx
</code></pre>
<p>I require applying a <strong>custom attention mask</strong> (not the causal one above) <strong>that is computed based on the specific sequence of tokens themselves</strong> (that is how my custom architecture works). So in a batch of 4 sequences, I might have 4 different masks, each depending on the sequence itself. I compute the masks outside this function, and store them in a tensor, and because this mask_mod function can read any tensor on the device, I can have a custom mask based on the sequence.<br>
But…</p>
<p>I initially assumed that “b” is the batch index, but its not.<br>
Without knowing which batch I am in, I cannot compute my own mask for each sequence. I am left with applying the same mask for the entire batch, and thus I cannot use Flex Attention to implement my architecture at scale, that is, using vLLM, with large batches.</p>
<p>I got quite far in patching the vLLM code and using the newly added Flex Attention backend. But I run into that same issue. vLLM works fenominaly well on batches of sequences. And because of the flaw with Flex Attention (not being able to set a mask for each sequence), I am restricted to maybe doing one sequence at a time in vLLM.</p>
<p>Could someone help me confirm that this indeed is a limitation that exists, and if so, will there be plans to fix this?<br>
Because I think if indeed I am correct, it makes Flex Attention not very useful for a wide range of architectures. We want to be able to have a custom attention mask, but not just “blind” attention patterns like checkerboard or sliding windows. Those patterns seldom make any difference. What is much more interesting are patterns <strong>based on the content of the sequence themselves</strong>, which is what I am trying to achieve.</p>
<p>Thank you!</p>
Libtorch, Cannot find NVTX3, find old NVTX instead

username: weitaoliu
cooked: <p>Hey,</p>
<p>I am compiling (using CMake 3.28) a CPP solver that links to libtorch (2.5.1). However, there is a warning about NVTX3 as "-- Could NOT find nvtx3 (missing: nvtx3_dir) "</p>
<p>I find it comes from libtorch/share/cmake/Caffe2/public/cuda.cmake</p>
<pre><code class="lang-auto">find_path(nvtx3_dir NAMES nvtx3 PATHS "${PROJECT_SOURCE_DIR}/third_party/NVTX/c/include" NO_DEFAULT_PATH)
find_package_handle_standard_args(nvtx3 DEFAULT_MSG nvtx3_dir)
if(nvtx3_FOUND)
  add_library(torch::nvtx3 INTERFACE IMPORTED)
  target_include_directories(torch::nvtx3 INTERFACE "${nvtx3_dir}")
  target_compile_definitions(torch::nvtx3 INTERFACE TORCH_CUDA_USE_NVTX3)
else()
  message(WARNING "Cannot find NVTX3, find old NVTX instead")
  add_library(torch::nvtoolsext INTERFACE IMPORTED)
  set_property(TARGET torch::nvtoolsext PROPERTY INTERFACE_LINK_LIBRARIES CUDA::nvToolsExt)
endif()
</code></pre>
<p>I checked “{PROJECT_SOURCE_DIR}” variable, and it refers to my source codes directory (where .c files are located), and of course, there is no “third-party” folder.</p>
<p>Configuration:<br>
libtorch version: stable 2.5.1<br>
CUDA version: 12.4</p>
<p>Do you have ideas on how to address this warning and check where NVTX3 is in the computer? Thanks for your time.</p>
<p>Best, Weitao.</p>

username: weitaoliu
cooked: <p>Hey guys,</p>
<p>I found a solution to this problem. You can download NVTX3 from <a href="https://github.com/NVIDIA/NVTX" class="inline-onebox" rel="noopener nofollow ugc">GitHub - NVIDIA/NVTX: The NVIDIA® Tools Extension SDK (NVTX) is a C-based Application Programming Interface (API) for annotating events, code ranges, and resources in your applications.</a>. Then, the directory pathway in libtorch/share/cmake/Caffe2/public/cuda.cmake should be adjusted correspondingly. Once you have the NVTX3, the warning will not appear.</p>
<p>Best, Weitao.</p>

username: 5had3z
cooked: <p>For at least pytorch 2.7 (where I encountered this problem), I noticed there’s a flag to use NVTX already installed on your system, enabling with <code>-DUSE_SYSTEM_NVTX:BOOL=ON</code> during cmake configure stage will fix this without needing to download anything else or change internal pytorch cmake code. This will then use the NVTX already included with CUDA Toolkit.</p>
<p><code>torch/share/cmake/Caffe2/public/cuda.cmake#L173</code></p>
<pre data-code-wrap="cmake"><code class="lang-cmake"># nvToolsExt
if(USE_SYSTEM_NVTX)
  find_path(nvtx3_dir NAMES nvtx3 PATHS ${CUDA_INCLUDE_DIRS})
else()
  find_path(nvtx3_dir NAMES nvtx3 PATHS "${PROJECT_SOURCE_DIR}/third_party/NVTX/c/include" NO_DEFAULT_PATH)
endif()
</code></pre>
_inductor/schduler.py line#3161

username: jongseoKang
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/5/55232f6de40803fce2cb5ca3974e8fb4004a737f.png" data-download-href="https://discuss.pytorch.org/uploads/default/55232f6de40803fce2cb5ca3974e8fb4004a737f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55232f6de40803fce2cb5ca3974e8fb4004a737f_2_690x220.png" alt="image" data-base62-sha1="c99Y59wUMIneg2KF6rWWZEBHHaL" width="690" height="220" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55232f6de40803fce2cb5ca3974e8fb4004a737f_2_690x220.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/5/55232f6de40803fce2cb5ca3974e8fb4004a737f_2_1035x330.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/5/55232f6de40803fce2cb5ca3974e8fb4004a737f.png 2x" data-dominant-color="F3F2EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1242×396 45 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hi, i’m undergraduate student interested in pytorch2.0’s inductor.<br>
While reading the code of operaotr fusing, i’m curious about line 3161 of schduler.py.</p>
<p>why node2’s dep footprint is evaluated with node1.read_writes?<br>
Is it just a simple typo?</p>
GradScaler decrease the scale suddenly on AMP multi-card training

username: JohnHerry
cooked: <p>OS: ubuntu_22.04<br>
python： 3.10.12<br>
CUDA： H800, CUDA:12.1<br>
torch: 2.4.1+cu121<br>
quesiton detail:<br>
This is maybe a bug for amp.GradScaler? I am not sure.<br>
I have a model for AMP training. When it is trained with a higher lr [eg. 1e-3] on a single card, it runs fine. but when I train it on a multiple card runing, it will get None grad.<br>
I had followed the scale value of the GradScaler (init with default), the value can be jumped in some step from 131072.0 to 5.29xxxx e-23, in just one step. accordding to introduction, when some step get Inf grad value, the GradScaler will shuink the scale value to half, that means when it change from 131072.0 to a very small float value, there should be many many training steps. How can this changed happend in just one step? Is this a bug? or some of my misunderstanding?</p>

username: ptrblck
cooked: <p>Do you see the same issue using a newer PyTorch release, e.g. 2.7.1 or a nightly binary?</p>
What does the "triaged" label mean in PyTorch issues?

username: xuantengh
cooked: <p>In some PyTorch GitHub issues, there is the “triaged” label attached.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/7/47397c65a648799612517fd872c6c700fc24ad14.png" data-download-href="https://discuss.pytorch.org/uploads/default/47397c65a648799612517fd872c6c700fc24ad14" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/7/47397c65a648799612517fd872c6c700fc24ad14.png" alt="image" data-base62-sha1="aa53VPfd3AxF6xKlKDvOx2bHgPy" width="466" height="138"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">466×138 5.81 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The label description is:</p>
<blockquote>
<p>This issue has been looked at a team member, and triaged and prioritized into an appropriate module</p>
</blockquote>
<p>Does it mean the issue is already being handled by someone, and can I should skip this issue to avoid conflict or duplicated works?</p>

username: Jack_Qasim
cooked: <p>The triaged label in PyTorch issues means the issue has been reviewed and categorized for further action.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="xuantengh" data-post="1" data-topic="221276">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/xuantengh/48/77571_2.png" class="avatar"> xuantengh:</div>
<blockquote>
<p>should skip this issue to avoid conflict or duplicated works</p>
</blockquote>
</aside>
<p>No, you can still work on this issue but should discuss it with the module owners after the issue was triaged in case someone else is already working on it.</p>
Using system NCCL when building PyTorch from source

username: BigFeetTinyBrain
cooked: <p>I’m trying to build PyTorch from source on an Ubuntu machine in a miniconda environment.</p>
<p>I’m trying to use the existing NCCL libraries installed on the system. I set the USE_SYSTEM_NCCL=1 but it still compiles and links with the NCCL submodule in the PyTorch repository.</p>
<p>I took a look at this issue - <a href="https://github.com/pytorch/pytorch/issues/32286" class="inline-onebox" rel="noopener nofollow ugc">Use system NCCL library in PyTorch · Issue #32286 · pytorch/pytorch · GitHub</a> which states that this is expected behavior. Is there any way to configure the build to use the system NCCL despite being built in a conda env? Or should I just build manually without conda? I don’t want to mess up any dependencies other users may have since this is a machine accessed by several users.</p>

username: ptrblck
cooked: <p>The env variable worked in the past properly and is now not needed anymore as we removed the <code>third_party/nccl</code> folder and now depend on the system NCCL installation. You could thus simply update your repository and rebuild as it’s the default behavior now.</p>

username: BigFeetTinyBrain
cooked: <p>Thanks for the reply! I checked out the v2.7.1 commit of PyTorch (e2d141db) before I ran the build and yet during my build. it pulled nccl into the third party folder. Do you think I’m doing something wrong? Please let me know if you need more specifics on my environment.</p>

username: BigFeetTinyBrain
cooked: <p>I now see that there is a function called git_checkout_nccl() which is called unconditionally in build_pytorch_libs in commit e2d141db that I have checked out. However, the later commits have added a check for the USE_SYSTEM_NCCL and USE_NCCL flags before calling it.</p>
<p>Additionally setting USE_SYSTEM_NCCL and USE_NCCL in my shell environment instead of in setup.py seems to correctly initialize the variables in my build.</p>
<p>This seems to have fixed my issue <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>Great! Thanks for confirming these env variables work for you, too!</p>
Multi-Head Self Attention in Transformer is permutation-invariant or equivariant how to see it in practice?

username: EricF
cooked: <p>I read that a function f is equivariant if f(P(x)) = P(f(x)) where P is a permutation</p>
<p>So to check what means equivariant and permutation invariant I wrote the following code</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

multihead_attn = nn.MultiheadAttention(embed_dim=32, num_heads=4, batch_first=True)
x0 = torch.ones(11,32)
x1 = torch.ones(11,32)
for i in range(x0.size(0)):
    x0[i] *= i
    x1[i] *= (i+1) % x0.size(0)

x = torch.cat(
    (x0.unsqueeze(0), x1.unsqueeze(0))
    )

y0, y1 = multihead_attn(x,x,x)[0]
y0 = y0.squeeze(0)
y1 = y1.squeeze(0)

</code></pre>
<p>Then to check torch.equal(x0[1],x1[0]) &gt;&gt;&gt; True</p>
<p>but torch.equal(y0,y1) &gt;&gt;&gt; False, so doesn’t seem to be permutation invariant<br>
and torch.equal(y0[1],y1[0]) &gt;&gt;&gt; False, so doesn’t seem to be equivarient</p>
<p>So what am I doing wrong ?</p>

username: paulk
cooked: <p>The issue is that you are using <code>torch.equal</code> on a float tensor. You can use<br>
<code>torch.allclose(y0[1], y1[0], atol=1e-6)</code> instead (which evaluates to <code>True</code>).<br>
I am not entirely sure what you are trying to do here but note that<br>
<code>print(torch.equal(x0[0], x1[1]))</code><br>
is <code>False</code> as well.<br>
Hope that helps</p>

username: EricF
cooked: <p><a class="mention" href="/u/paulk">@paulk</a><br>
Thanks a lot, yes that perfectly answers my question. It shows that multi head attention is equivariant and not permutation invariant as I often read.</p>
<p>I do not understand why, but on my computer <code>print(torch.equal(x0[0], x1[1]))</code>is True</p>

username: paulk
cooked: <p>Glad it helped. Not sure if this is relevant then but when I run:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

multihead_attn = nn.MultiheadAttention(embed_dim=32, num_heads=4, batch_first=True)
x0 = torch.ones(11,32)
x1 = torch.ones(11,32)
for i in range(x0.size(0)):
    x0[i] *= i
    x1[i] *= (i+1) % x0.size(0)

x = torch.cat(
    (x0.unsqueeze(0), x1.unsqueeze(0))
    )

y0, y1 = multihead_attn(x,x,x)[0]
y0 = y0.squeeze(0)
y1 = y1.squeeze(0)

print(torch.equal(x0[0], x1[1]))
</code></pre>
<p>It prints <code>False</code> while</p>
<pre><code class="lang-auto">print(torch.equal(x0[1], x1[0]))
</code></pre>
<p>prints <code>True</code>.<br>
But that makes sense and your rotation works fine (for some reason I assumed x had shape [2, 32] instead of [11, 32] and thus thought <code> torch.equal(x0[0], x1[1])</code> should be <code>True</code> as well.)</p>
<p><strong>TLDR:</strong><br>
Don’t worry about it, my bad.</p>

username: EricF
cooked: <p>You are totaly right. It was a type error <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=14" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"><br>
Many thanks</p>
Is there a memory mechanism in PyTorch similar to zram?

username: chenghua_wang
cooked: <p>In operating systems, zram is commonly used to compress memory in order to reduce memory usage. I’m wondering if PyTorch has a similar mechanism for optimizing memory consumption.</p>

username: William_Gram
cooked: <p>PyTorch doesn’t offer a zram-style memory compression mechanism. Instead, it optimizes GPU RAM by using a <strong>CUDA caching allocator</strong>:</p>
<ul>
<li><strong>Memory pooling</strong>: it caches freed GPU memory blocks for reuse—avoiding frequent <code>cudaMalloc</code>/<code>cudaFree</code> calls and fragmentation, which speeds things up (<a href="https://github.com/pytorch/pytorch/issues/124807?utm_source=chatgpt.com" title="[RFC] Mix and Match CUDA Allocators using Private Pools · Issue #124807 · pytorch/pytorch · GitHub" rel="noopener nofollow ugc">github.com</a>).</li>
<li><strong>Use custom allocators or memory pools</strong>: advanced users can plug in their own allocators (<code>CUDAPluggableAllocator</code>) or use <code>torch.cuda.MemPool</code> alongside caching (<a href="https://github.com/pytorch/pytorch/issues/124807?utm_source=chatgpt.com" title="[RFC] Mix and Match CUDA Allocators using Private Pools · Issue #124807 · pytorch/pytorch · GitHub" rel="noopener nofollow ugc">github.com</a>).</li>
</ul>
<p>So, rather than zram compression of RAM, PyTorch focuses on efficient reuse and tenancy of GPU memory blocks.</p>
DTensor TP collectives missing?

username: githubsgi
cooked: <p>This  <a href="https://docs.pytorch.org/tutorials/intermediate/TP_tutorial.html" rel="noopener nofollow ugc">doc</a>  and the <a href="https://arxiv.org/pdf/1909.08053" rel="noopener nofollow ugc">paper on TP</a>  mentions the following.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc338f1b363b86c24806e324b347e325144f17a4.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/dc338f1b363b86c24806e324b347e325144f17a4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dc338f1b363b86c24806e324b347e325144f17a4_2_434x500.jpeg" alt="image" data-base62-sha1="vpZvvockezDVWX3q4Ke2yOHnZD6" width="434" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/c/dc338f1b363b86c24806e324b347e325144f17a4_2_434x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc338f1b363b86c24806e324b347e325144f17a4.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/c/dc338f1b363b86c24806e324b347e325144f17a4.jpeg 2x" data-dominant-color="DDE5DF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">501×577 82.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Figure 3. Blocks of Transformer with Model Parallelism. f and g are conjugate. f is an identity operator in the forward pass and all reduce in the backward pass while g is an allreduce in the forward pass and identity in the backward pass.</p>
<p>However, <a href="https://github.com/pytorch/examples/blob/main/distributed/tensor_parallelism/tensor_parallel_example.py" rel="noopener nofollow ugc">running this TP example</a> of MLP, I can only see 1 allreduce  corresponding to the forward pass. Do not see any allreduce corresponding to the backward pass.   Can someone clarify what is happening in the DTensor TP ?</p>
Why pytorch use triton as backend of inductor?

username: Donald
cooked: <p>The behavior of Inductor using Triton is quite puzzling to me. I’d really appreciate any insights on this matter.</p>
<p><strong>Why triton Language</strong></p>
<p>In pursuit of higher performance and more fine-grained control over hardware, most compilers opt to generate lower-level IR rather than a language, unless that language is extremely optimized for the hardware (such as TVM generating CUDA C). In most cases, the code generated by the compiler does not require manual modification at the language level. I would like to understand what factors led PyTorch to choose the Triton language as the lowering target for Inductor.</p>
<p><strong>What’s next</strong><br>
Another thing I’m curious about is: what are the criteria for evaluating and selecting a backend language? Triton demonstrates strong expressiveness and performance in large language models, but in other domains, its limitations and lack of expressiveness are currently quite apparent. In terms of performance and expressiveness, both mojo and cu-tile seem to have the potential to surpass Triton. In the future, under what circumstances might Inductor consider replacing its backend language or compiler?</p>
<p>Thank you for all the responses related to this topic.</p>

username: Donald
cooked: <p><a class="mention" href="/u/leslie-fang-intel">@leslie-fang-intel</a> <a class="mention" href="/u/smth">@smth</a> ping</p>

username: Donald
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/richard">@richard</a> ping</p>
Update NCCL timeout

username: Sam_Shleifer
cooked: <p>I want to update the nccl timeout from 30 minutes to 5 minutes after step 1</p>
<pre><code>        torch.distributed.init_process_group(
            backend="nccl",
            world_size=world_size,
            rank=rank,
            timeout=timedelta(minutes=30),  # 30 is the default but be explicit
        )
</code></pre>
<p>Is that possible somehow? Maybe there is an os.environ trick?</p>
In-place modification in backward not allowed with torch.compile

username: Julian_Buchel
cooked: <p>Modifying the gradient in-place does not work in torch.compile. If I clone the incoming gradient, zero out the elements I want to zero out, and then passing that on works. But doesn’t that use too much memory?</p>
<p>What could be a way to achieve this without blowing up memory but still being able to use torch.compile?</p>
<pre><code class="lang-auto">class StraightThroughClamp(Function):
    """
    Straight-through clamp function.
    """

    # pylint: disable=abstract-method, redefined-builtin, arguments-differ

    @staticmethod
    def forward(
        ctx: FunctionCtx,
        x_input: Tensor,
        upper_thresh: Tensor,
        lower_thresh: Tensor,
        input_range: Tensor,
    ) -&gt; Tensor:
        clamped_mask = upper_thresh | lower_thresh
        exactly_the_same = x_input.abs() == input_range
        ctx.save_for_backward(clamped_mask &amp; ~exactly_the_same)
        return clamp(x_input, min=-input_range, max=input_range)

    @staticmethod
    # type: ignore[override]
    def backward(ctx: FunctionCtx, d_output: Tensor) -&gt; Tuple[Tensor, None, None, None, None]:
        (clamped_mask,) = ctx.saved_tensors  # type: ignore[attr-defined]
        
        # doesn't this blow up memory?
        d_output_zeroed_out = d_output.clone()
        d_output_zeroed_out[clamped_mask] = 0.0

        # what works, but not when I use torch.compile
        d_output[clamped_mask] = 0.0

        return d_output_zeroed_out, None, None, None, None
</code></pre>

username: soulitzer
cooked: <p>Are you using the latest version of PyTorch? We added support to allow mutating tangents in compile here <a href="https://github.com/pytorch/pytorch/pull/141131" class="inline-onebox" rel="noopener nofollow ugc">support input mutations on tangents in compile by bdhirsh · Pull Request #141131 · pytorch/pytorch · GitHub</a>.</p>
<p>I would say that in-place modification of gradients is actually unsafe in eager today, for example if you did an add afterwards, the gradient you are receiving will be aliased to another tensor. There are plans to add safe guards to allow doing this safely, however.</p>
Scoring batch samples using MultivariateNormal

username: HDubois-fr
cooked: <p>Hi,<br>
I have a set of k MultivariateNormal distributions in d dimension.</p>
<pre><code class="lang-auto">mu = torch.FloatTensor(k, d)
sigma = torch.FloatTensor(k, d, d)
...
D = torch.distributions.MultivariateNormal(loc=mu, scale_tril=sigma)
</code></pre>
<p>I have a batch of N d-dimensionnal samples, and I want to get the log_prob for each of the distributions (so k values per sample).</p>
<p>When I do</p>
<pre><code class="lang-auto">D.log_prob(batch)
</code></pre>
<p>With N = 64, k=8 and d=16, I get the following error :</p>
<pre><code class="lang-auto">ValueError: Value is not broadcastable with batch_shape+event_shape: torch.Size([64, 16]) vs torch.Size([8, 16]).
</code></pre>
<p>I’ve tried expanding the distribution to match the batch shape but I get the error is equivalent:</p>
<pre><code class="lang-auto">ValueError: Value is not broadcastable with batch_shape+event_shape: torch.Size([64, 16]) vs torch.Size([64, 8, 16]).
</code></pre>
<p>Is there a way to do it that doesn’t involve iterating through the batch and calling log_prob N times ?</p>

username: tbloch1
cooked: <p>For anyone else trying to solve this, the trick is that you have to keep the batch and shape dimensions on the right. E.g., in the example from <a class="mention" href="/u/hdubois-fr">@HDubois-fr</a>, it would work if <code>batch</code> was shaped [N, k, d].</p>
<p>I couldn’t find much that explained this. It works with arbitrary N shapes too making various calculations easier for you. E.g., In the above example a shape of [N,M,k.d] would also work fine.</p>
How to check device of torch.nn.Module?

username: RylanSchaeffer
cooked: <p>I received the following error:</p>
<pre><code class="lang-auto">File ".../lib/python3.6/site-packages/torch/nn/functional.py", line 1852, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Input, output and indices must be on the current device
</code></pre>
<p>So I wanted to check what devices the three variables were on. For the tensors, I could use <code>tensor.get_device()</code> and that worked fine. However, when I tried checking what device the problematic <code>torch.nn.Module</code> was on, I received the error:</p>
<p><code>torch.nn.modules.module.ModuleAttributeError: 'DistilBertForMaskedLM' object has no attribute 'get_device'</code></p>
<p>How does one check the device of a <code>torch.nn.Module</code> object?</p>

username: RylanSchaeffer
cooked: <p>I just found this Stack Overflow post. I understand now.</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/58926054/how-to-get-the-device-type-of-a-pytorch-module-conveniently">
  <header class="source">

      <a href="https://stackoverflow.com/questions/58926054/how-to-get-the-device-type-of-a-pytorch-module-conveniently" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/12128185/kani" target="_blank" rel="noopener nofollow ugc">
    <img alt="Kani" src="https://i.stack.imgur.com/Fp3jc.png?s=128&amp;g=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/58926054/how-to-get-the-device-type-of-a-pytorch-module-conveniently" target="_blank" rel="noopener nofollow ugc">How to get the device type of a pytorch module conveniently?</a>
</h4>

<div class="tags">
  <strong>python, gpu, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/12128185/kani" target="_blank" rel="noopener nofollow ugc">
    Kani
  </a>
  on <a href="https://stackoverflow.com/questions/58926054/how-to-get-the-device-type-of-a-pytorch-module-conveniently" target="_blank" rel="noopener nofollow ugc">03:09AM - 19 Nov 19 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: mpeschel10
cooked: <p>For future readers, here is a summary of RylanSchaeffer’s link to StackOverflow:</p>
<p>Since a Module’s parameters can be on multiple different devices, there is no formal way to get the device the Module is on.</p>
<p>A “good enough” solution, if you know your model will only be on one device, is:</p>
<pre data-code-wrap="python"><code class="lang-python">next(model.parameters()).device
</code></pre>