akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How to embed pytorch profiler to verl?

gogineni_kailash: <p>This is my code base and I want to add pytorch profiler to the fit() function. Can anyone let me know how to embed the pytorch profiler?</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py">
  <header class="source">

      <a href="https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py" target="_blank" rel="noopener nofollow ugc">github.com/volcengine/verl</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py" target="_blank" rel="noopener nofollow ugc">verl/trainer/ppo/ray_trainer.py</a></h4>

<div class="git-blob-info">
  <a href="https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py" rel="noopener nofollow ugc"><code>main</code></a>
</div>


      <pre><code class="lang-py"># Copyright 2024 Bytedance Ltd. and/or its affiliates
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""
FSDP PPO Trainer with Ray-based single controller.
This trainer supports model-agonistic model initialization with huggingface
"""

import os
import uuid
</code></pre>



  This file has been truncated. <a href="https://github.com/volcengine/verl/blob/main/verl/trainer/ppo/ray_trainer.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I also raised the issue in the verl github, but was not resolved. Any help or insights will be appreciated.<br>
It only print the cpu events without CUDA events. But, the GPU utilization and VRAM is almost 80 to 100% over the course of training.</p>
<p>Issue: <a href="https://github.com/volcengine/verl/issues/693" class="inline-onebox" rel="noopener nofollow ugc">GitHub · Where software is built</a></p>
LSTM for TSC - effect of sequence length

eugenio_b: <p>I am developing a TSC classifier based on LSTM with a fully connected layer, below a portion of the code</p>
<blockquote>
<pre><code>def forward(self, x):
    if not self.get_is_stateful(): # stateless, reset hidden state at each forward pass
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(self.get_device())
        c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(self.get_device())
        out_lstm, _ = self.lstm(x, (h0,c0))
    else: # stateful, keep hidden state between forward passes
        h0 = self.hidden_state.detach().clone()
        c0 = self.cell_state.detach().clone()
        out_lstm, (self.hidden_state, self.cell_state) = self.lstm(x, (h0, c0))      
    out = self.fc(out_lstm[:, -1, :]) if self.get_batch_first() else self.fc(out_lstm[-1, :, :]) 
</code></pre>
</blockquote>
<p>What happens if I train the model with the whole sequence instead of passing the input timestamp per timestamp? From the documentation, I understand the output will have one dimension equal to the length of the sequence, but in this case: 1) the forward method would be called for each timestamp sequentially (and as a consequence hidden state will be the input of the next timestamp) or in parallel? 2)The only thing that I am quite sure is that there is only one model and not lenght(sequence) model in parallel</p>

J_Johnson: <p>In the case of PyTorch LSTM, you can pass in the entire sequence and it will handle iterating through it under the hood. As I understand it, there are some tricks used which make it faster than doing so manually, one token at a time.</p>
<p>Regarding feeding the model timestamps, I assume you meant positional encoding? As far as I know, no study has been done that has shown any benefit (or lack there of) for adding positional encoding to a recurrent neural network model(i.e. LSTM). That’s only been demonstrated to work in Transformers. It’s believed that sequential information is learned and held within the hidden state, and updated at each token pass.</p>

eugenio_b: <p>Ok, it will handle iterating but the LSTM pytorch module (not mine forward implementation) I guess, so how can I force pytorch lstm module to work as a stateful lstm (so by using previous hidden and cell state output for the current step, for each step in the sequence)?</p>
Questions about using the reduce_scatter api in nccl.h. How should the input and output parameters in the api be defined?

sushe: <p>Questions about using the reduce_scatter api in nccl.h. How should the input and output parameters in the api be defined?<br>
Check the code: On line 752, const auto count = inputs[0].numel() / len;<br>
In ncclReduceScatter, this count is required to be recv_count.   They don’t seem to match.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/torch/csrc/cuda/nccl.cpp#L739">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/torch/csrc/cuda/nccl.cpp#L739" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/torch/csrc/cuda/nccl.cpp#L739" target="_blank" rel="noopener nofollow ugc">torch/csrc/cuda/nccl.cpp</a></h4>

<div class="git-blob-info">
  <a href="https://github.com/pytorch/pytorch/blob/main/torch/csrc/cuda/nccl.cpp#L739" rel="noopener nofollow ugc"><code>main</code></a>
</div>



    <pre class="onebox"><code class="lang-cpp">
      <ol class="start lines" start="729" style="counter-reset: li-counter 728 ;">
          <li>        data_type,</li>
          <li>        to_nccl_red_op(op),</li>
          <li>        to_nccl_comm(comm),</li>
          <li>        stream));</li>
          <li>  }</li>
          <li>#else</li>
          <li>  TORCH_CHECK(false, "PyTorch built without NCCL support");</li>
          <li>#endif</li>
          <li>}</li>
          <li></li>
          <li class="selected">void reduce_scatter(</li>
          <li>    const std::vector&lt;at::Tensor&gt;&amp; inputs,</li>
          <li>    std::vector&lt;at::Tensor&gt;&amp; outputs,</li>
          <li>    int32_t op,</li>
          <li>    const stream_list&amp; streams,</li>
          <li>    const comm_list&amp; user_comms) {</li>
          <li>#ifdef USE_NCCL</li>
          <li>  using namespace torch::cuda::nccl::detail;</li>
          <li>  const auto len = inputs.size();</li>
          <li>  check_inputs(inputs, outputs, 1, len);</li>
          <li></li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


neoncube: <p>It’s tricky. I found this tutorial to be <em>extremely</em> helpful when I was learning <code>scatter()</code>: <a href="https://yuyangyy.medium.com/understand-torch-scatter-b0fd6275331c" class="inline-onebox" rel="noopener nofollow ugc">Understand torch.scatter_(). First, note that scatter_() is an… | by Yu Yang | Medium</a></p>

sushe: <p>Thank you for your reply. My question is about the NCCL communication library, and it may be different from yours.</p>

neoncube: <p>Yes, you’re right, sorry <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Unexpected CUDA IPC handle behaviour

Gino: <p>Hello, I’m currently trying to copy into a pytorch tensor the data from a c++ process. Since the data is also on the GPU and accessible through CUDA, I was thinking of using a simple cudaMemcpy to achieve efficient results.</p>
<p>The problem is that I’d need a CUDA accessible GPU address.<br>
Reading <a href="https://discuss.pytorch.org/t/using-cuda-ipc-memory-handles-in-pytorch/17548">this</a> previous post, it mentions that using storage._share_cuda_() gives you access to the cudaIpcMemhandle_t, which in theory I could use either on the python side (through libraries like pycuda), or on the c++ to get the device pointer.</p>
<p>The issues I’m having is that firstly is that I’m getting a 66 bytearray instead of a 64 byte array, which is cudaIpcMemhandle_t’s actual size. Secondly is that even when I arbitrarly choose the last 64 byte and try to use pycuda’s IPCMemoryHandle function, I get a invalid device context error.<br>
When I try to open the handle in c++ it doesn’t work directly.</p>
<p>I’m quite unfamiliar with pytorch so any help would be appreciated.</p>

Gino: <p>Managed to get the issue fixed. Basically I screwed up the c++ code.<br>
I also got it to work by simply creating an ipcmemhandle from the device pointer of the tensor as well (through debugging I got that)</p>
Resnet18 behaving differently in libtorch than pytorch

Youssef_Mohamed: <p>So i trained resnet18 on a dataset in python and got 100% accuracy i made an infrence code in python and c++ using libtorch both libtorch and pytorch are same versions on windows.<br>
They both used jit.load and loaded sucessfully and the input matrix in the forward pass for both were identical for the same image i even enusred the subtle floating point was exactly the same however in libtorch before the soft max the output is thousands and introduces wrong classifications as a result and the output in pytorch is 5. Smth.</p>
<p>Please note that this is not an issue with preprocessing or data collection since the input matrix and library versions between the two are 2.1.2 both python and cpp additionally the outputs collected are also in the same shape.</p>

azhanmohammed: <p>Just to be double sure, could you dump the input matrix for both c++ and python before feeding the inputs to the model.</p>
Very small, stupid question about FSDPParam._init_sharded_param

byi8220: <p>I wasn’t sure where to ask about this since this is such a minor question about FSDP internals.</p>
<p>In FSDP2’s <a href="https://github.com/pytorch/pytorch/blob/3e0038ae85246c1d3ffd95a618bb8fdebd9dd513/torch/distributed/fsdp/_fully_shard/_fsdp_param.py#L259" rel="noopener nofollow ugc"><code>FSDPParam._init_sharded_param</code></a>, this function instantiates a <code>nn.Parameter</code> object and assigns it to <code>self.sharded_param</code>, then calls its setter <code>requires_grad_</code> on it:</p>
<pre data-code-wrap="python"><code class="lang-python"># From: https://github.com/pytorch/pytorch/blob/3e0038ae85246c1d3ffd95a618bb8fdebd9dd513/torch/distributed/fsdp/_fully_shard/_fsdp_param.py#L398-L399
self.sharded_param = nn.Parameter(self.to_sharded_dtensor(sharded_param))
self.sharded_param.requires_grad_(param.requires_grad)
</code></pre>
<p><a href="https://github.com/pytorch/pytorch/blob/3e0038ae85246c1d3ffd95a618bb8fdebd9dd513/torch/nn/parameter.py#L40" rel="noopener nofollow ugc"><code>nn.Parameter.__new__</code></a> takes in <code>requires_grad</code> as an argument, defaulting it to <code>True</code>.</p>
<p>Suppose <code>sharded_param</code> is an integer type, for example, <code>torch.uint8</code> for quantization purposes. Then we first attempt to create a parameter with <code>requires_grad=True</code> in order to set it to false immediately afterwards. This fails, since only float type parameters can have a gradient.</p>
<p>Would <code>self.sharded_param = nn.Parameter(self.to_sharded_dtensor(sharded_param), requires_grad=sharded_param.requires_grad)</code> be more accurate, or is there a reason this has to be separate?</p>
<p>I’m asking since I’m experimenting with fsdp2 and qlora and wondering if this is intentional.</p>
Control whether custom op outputs have requires_grad set

BeachHut: <p>For a custom operator, created using the currently recommended approach, is there a way to control whether each of the outputs has requires_grad set? It seems that by default all will be set to True if any of the inputs has requires_grad set, and otherwise all will be False. In my operator, whether each output actually requires gradients depends on whether certain inputs (a different set for each output) require them. I would like this to be accurately reflected, rather than for all of the outputs to have requires_grad set if any of the inputs do.</p>
C10d ipv6 network address cannot be retrieved error

arunppsg: <p>I am trying a multinode training using this example: <a href="https://github.com/pytorch/examples/blob/main/distributed/ddp-tutorial-series/multinode.py" rel="noopener nofollow ugc">https://github.com/pytorch/examples/blob/main/distributed/ddp-tutorial-series/multinode.py</a> . My infrastructure setup is two AWS g4dn.xlarge instances, each with single GPU and I invoked the scripts via <code>torchrun</code>.</p>
<p>On node 0, the script is invoked as <code>torchrun --nproc-per-node=1 --nnodes=2 --node-rank=0 --rdzv-id=456 --rdzv-backend=c10d --rdzv-endpoint=172.16.130.32:16000 multinode.py 10 5</code> and on node 1, as <code>torchrun --nproc-per-node=1 --nnodes=2 --node-rank=1 --rdzv-id=456 --rdzv-backend=c10d --rdzv-endpoint=172.16.130.32:16000 multinode.py 10 5</code></p>
<p>Here, <code>172.16.130.32</code> is the private ipv4 address of node0.</p>
<p>When I do the above, I get a error log as</p>
<pre><code class="lang-auto">master_addr is only used for static rdzv_backend and when rdzv_endpoint is not specified.
[W socket.cpp:601] [c10d] The IPv6 network addresses of (ip-172-16-130-220, 52737) cannot be retrieved (gai error: -3 - Temporary failure in name resolution).
[W socket.cpp:601] [c10d] The IPv6 network addresses of (ip-172-16-130-220, 52737) cannot be retrieved (gai error: -3 - Temporary failure in name resolution).
</code></pre>
<p>in node 0.</p>
<p><code>172.16.130.220</code> is the private ipv4 of node 2.</p>
<p>Any hints on resolving it will be helpful.</p>

arunppsg: <p>I didn’t enable DNS Resolution and DNS hostname in AWS VPC. After enabling them, it worked.</p>

Kokul_Raj: <p>I am using Mac M2, have following error<br>
torch.distributed.elastic.multiprocessing.redirects: [WARNING] NOTE: Redirects are currently not supported in Windows or MacOs.<br>
[W socket.cpp:697] [c10d] The IPv6 network addresses of (1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa, 51738) cannot be retrieved (gai error: 8 - nodename nor servname provided, or not known).</p>

nrxsvzo: <p>For others that encounter this error, this blog may be helpful: <a href="https://hychiang.info/blog/2024/pytorch_distributed_training/" class="inline-onebox" rel="noopener nofollow ugc">Distributed Training with Pytorch | Hung-Yueh Chiang</a></p>
Selective gradient clipping

SudakshK: <p>Hi,<br>
I have been working on an application where I need to partly freeze the layers and I do this by masking gradients to zero in each epoch for particular weights.<br>
Thus I removed gradient clipping while training vit-b-16 model on imagenet, as it always changed my gradeints which need to be 0. It turned out that the gradients were exploding and I recieved nan loss. How can I avoid this, while keeping gradient from not exploding and also freeze the grads. Is it possible to selectively apply grad clipping on particular weights?</p>

ptrblck: <aside class="quote no-group" data-username="SudakshK" data-post="1" data-topic="218493">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sudakshk/48/75478_2.png" class="avatar"> SudakshK:</div>
<blockquote>
<p>Is it possible to selectively apply grad clipping on particular weights?</p>
</blockquote>
</aside>
<p>Yes, you can pass a subset to the clipping operator:</p>
<pre data-code-wrap="python"><code class="lang-python">model = models.resnet18()
params_for_grad_clipping = itertools.chain(model.conv1.parameters(), model.fc.parameters())

x = torch.randn(1, 3, 224, 224)
out = model(x)
out.mean().backward()

print(model.conv1.weight.grad.abs().sum(), model.fc.weight.grad.abs().sum())
# tensor(42.4067) tensor(420.8640)

torch.nn.utils.clip_grad_value_(params_for_grad_clipping, clip_value=1e-7)
print(model.conv1.weight.grad.abs().sum(), model.fc.weight.grad.abs().sum())
# tensor(0.0009) tensor(0.0512)
</code></pre>

SudakshK: <p>Hi Patrick,<br>
Really thanks for the reply, but maybe I will rephrase my question, I need to partially freeze each layer. Below is the code snippet:</p>
<pre><code class="lang-auto">with torch.no_grad():
                layer.weight.grad[freeze_dim0, freeze_dim1] = 0
                if layer.bias is not None:
                        layer.bias.grad[freeze_dim0] = 0
</code></pre>
<p>So I am freezing specific parts of the weights and not whole weight, can I apply gradient clipping only on the remaining weights, whose gradients are still non-zero?</p>

ptrblck: <p>I don’t think this would be directly possible without workarounds of creating custom trainable and frozen weights in custom modules.<br>
Given that gradient clipping should not change gradients with a value of zero you should be able to just apply it on the entire parameter.<br>
If you want to compute e.g. the <code>norm</code> of the trainable part only, you could use <a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.clip_grads_with_norm_.html#torch.nn.utils.clip_grads_with_norm_"><code>torch.nn.utils.clip_grads_with_norm_</code></a> and pass the pre-calculated <code>total_norm</code> to this call.</p>

SudakshK: <p>Thanks! Got it, It works in this way!</p>
.item() blocks cpu thread until D2H operations in offload stream finish

zhr2001: <p>When using the following code to overlap H2D (host-to-device) and D2H (device-to-host) operations in an offload stream with operations in other streams, profiling reveals that <code>item()</code> blocks CPU threads until the last D2H operation completes. This behavior seems unexpected because <code>cudaStreamSynchronize</code> should only synchronize the current stream and shouldn’t affect other streams.</p>
<p>I’d like to understand:</p>
<ol>
<li>What causes this synchronization behavior?</li>
<li>How can I avoid this unintended synchronization?</li>
</ol>
<p>Here’s my test code:</p>
<pre data-code-wrap="python"><code class="lang-python">offload_stream = torch.cuda.Stream()

def test_stream():
    tensor_gpu = torch.ones((16384, 16384), device='cuda')
    tensor_cpu = torch.empty_like(tensor_gpu, device='cpu', pin_memory=True)

    seqlens = torch.ones((1,10240), device='cuda')
    a = torch.ones((10240,10240), device='cuda')  
    seqlens = torch.matmul(seqlens, a)
    

    for _ in range(20):
        with torch.cuda.stream(offload_stream):
            tensor_cpu.copy_(tensor_gpu, non_blocking=True)
            tensor_gpu.copy_(tensor_cpu, non_blocking=True)
            tensor_gpu.record_stream(offload_stream)
    
    for _ in range(20):
        seqlens = torch.matmul(seqlens, a)
    seqlens[0][0].item()

    torch.cuda.current_stream().wait_stream(offload_stream)
</code></pre>
<p>and trace:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/6/96c3048070329d6d38775dbb7b6e623ec88390eb.png" data-download-href="https://discuss.pytorch.org/uploads/default/96c3048070329d6d38775dbb7b6e623ec88390eb" title="截屏2025-04-08 下午12.48.34"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96c3048070329d6d38775dbb7b6e623ec88390eb_2_690x386.png" alt="截屏2025-04-08 下午12.48.34" data-base62-sha1="lvHrw1JX5bkRbTGXnmnNA63IRez" width="690" height="386" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96c3048070329d6d38775dbb7b6e623ec88390eb_2_690x386.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96c3048070329d6d38775dbb7b6e623ec88390eb_2_1035x579.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/6/96c3048070329d6d38775dbb7b6e623ec88390eb_2_1380x772.png 2x" data-dominant-color="BCB4C2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">截屏2025-04-08 下午12.48.34</span><span class="informations">2294×1284 377 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This test execute on <strong>NVIDIA-H20</strong> devices.</p>
<p>Thank you for your insights!</p>
Pytorch Dataloader Memory Leak

nmamie: <p>Hi, I noticed that while training a PyTorch model the subprocesses that are started by the dataloader workers are accumulating memory over time while loading new batches and it seems this memory is never released, ultimately resulting in a “dataloader worker does not have sufficient shared memory” error. Could this be a memory leak or is this a known “bug”.<br>
The PyTorch version i am using is 2.0 for CUDA 11.7 in case this information helps.</p>
<p>Thanks a lot!</p>

ptrblck: <p>Could you check if you are running into <a href="https://github.com/pytorch/pytorch/issues/13246#issuecomment-445446603">this issue</a>?</p>

nmamie: <p>Yes I already checked this issue, unfortunately this didn’t help. What I notice is that even if I set num_workers = 0 the shared memory of the process that runs the script keeps increasing until the shared memory error appears.</p>

ptrblck: <p>Shared memory shouldn’t be used of no multiprocessing is needed in the <code>DataLoader</code>s. Are you manually sharing tensors somewhere in your code?</p>

nlgranger: <p>I am also observing a shared memory leak in the parent process. It keeps growing way beyond the sum of shm in the dataloader workers.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/e/3e310a70c78b2c02c62fc6398de1b94964fb9ff7.png" alt="Capture d’écran du 2023-11-07 16-02-24" data-base62-sha1="8SaFHkgYN1uFP5DNMANSAoAGRBZ" width="496" height="189"></p>
<p>I also verified that it is <strong>not</strong> related to the dataloader.</p>

nlgranger: <p>Ok there seems to be a bug when runnning a whole script under a torch.inference_mode() context. Using inference mode only on model calls works fine and the leak is gone. I’ll try to make a minimal reproducible example.</p>

Winston_M: <p>I think I am encountering the same problem. my eval() function leaks memory</p>
<p>The eval() function is wrapped with torch.no_grad() and the model is set to model.eval()</p>
<p>Is this the same issue? Could you please point to my problem?</p>

nlgranger: <p>Hi! In my case, the problem was that I was saving my targets straight from the dataloader and thus with tensors stored in shared memory.</p>
<p>If you move that data to the GPU or unshare it should solve the issue.</p>
<pre data-code-wrap="py"><code class="lang-py">all_predictions = []
all_targets = []
for batch_in, batch_tgt in val_loader:
    batch_in = batch_in.to(device, non_blocking=True)
    bath_tgt = batch_tgt.clone()  # &lt;&lt;&lt;--- do this

    with torch.inference_mode():
        batch_prediction = model(batch_in)

    all_predictions.extend(batch_prediction)
    all_targets.extend(bath_tgt)
</code></pre>

0rhisia0: <p>+1 This solved my issue. Seems like anytime you don’t use a copy of the tensor returned by the loader, it defaults to writing it to shared memory (without garbage collecting it as needed)</p>
Compiling variable-size cross entropy

jjjj: <p>On both nightly and 2.6 the following code fails due to a data-dependent expression inside cross_entropy</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
torch._dynamo.config.capture_dynamic_output_shape_ops = True

@torch.compile(fullgraph=True)
def loss(logits, target, mask):
  logits = logits[mask, :]
  target = target[mask]
  return F.cross_entropy(logits, target)

loss(torch.rand(10, 4), torch.randint(0, 4, (10,)), torch.ones(10, dtype=torch.bool))
</code></pre>
<p>is it possible to fully compile this?</p>
How to setup buffers and parameters for mixed precision training

pytorcher: <p>I have a module which uses backwards hooks, buffers, and parameters, that works properly with either float of half tensors when specified. I am now using it with AMP mixed precision for the first time and it is causing this error, but I don’t know what to do to resolve it. I have set <code>torch.autograd.set_detect_anomaly(True)</code> which grants the following error:</p>
<pre><code class="lang-auto">ENVdavit/lib/python3.12/site-packages/torch/autograd/graph.py:823: UserWarning: 
Error detected in SigmoidBackward0. Traceback of forward call that caused the error:
  File
......
  File "myModule.py", line 878, in forward
    Outs[i] = torch.sigmoid(earlierOuts[i]).to(device)
 (Triggered internally at /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:122.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File line 827, in train_one_epoch
    loss_scaler(
  File "davit/timm/utils/cuda.py", line 43, in __call__
    self._scaler.scale(loss).backward(create_graph=create_graph)
  File "ENVdavit/lib/python3.12/site-packages/torch/_tensor.py", line 626, in backward
    torch.autograd.backward(
  File "ENVdavit/lib/python3.12/site-packages/torch/autograd/__init__.py", line 347, in backward
    _engine_run_backward(
  File "ENVdavit/lib/python3.12/site-packages/torch/autograd/graph.py", line 823, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
           ^^^^^^^^^^^

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: hook '&lt;lambda&gt;' has changed the type of value (was torch.cuda.HalfTensor got torch.cuda.FloatTensor)
</code></pre>
Torchrl kl_div for old and new policy

manavkoolz: <p>Hi,</p>
<p>I’m having some trouble understanding the ProbabilisticActor class, I think. Essentially, I want to monitor the KL divergence between the new and old policy. To do this, though multi-agent, my code is essentially this:</p>
<pre><code class="lang-auto"># ...
optim.step()
optim.zero_grad()

with torch.no_grad():
    old_log_prob = subdata['agents', 'action_log_prob']
    new_log_prob = policy(subdata.clone())['agents', 'action_log_prob']
    # kl_div(q.log(), p, log_target=False) -&gt; p * (p / q).log()
    kl = F.kl_div(new_log_prob, old_log_prob, log_target=True)
</code></pre>
<p>Is this the correct way to do this? Another thing that confuses me is that the docs for ProbabilisticActor say the output key will be <code>'sample_log_probs'</code> which does not match. Is this because I am specifying out_key? Though <code>env.action_key == ('agents', 'action')</code></p>
<pre><code class="lang-auto">policy = ProbabilisticActor(
    module=policy_module,
    spec=env.action_spec_unbatched,
    in_keys=[("agents", "loc"), ("agents", "scale")],
    out_keys=[env.action_key],
    distribution_class=TanhNormal,
    distribution_kwargs={
        "low": env.full_action_spec_unbatched[env.action_key].space.low,
        "high": env.full_action_spec_unbatched[env.action_key].space.high,
    },
    return_log_prob=True,
)
</code></pre>
Tackling Low GPU Kernel Occupancy During Loss Function Computation

dongyang2011: <p>Hello! We’re currently encountering a problem. Our loss function takes an extended period to compute, yet the occupancy of the GPU kernel is extremely low. We attempted to run each loss function in a separate process, but autograd isn’t capable of managing inter - process propagation. We’re wondering if there’s a way to parallelize the loss function. For example, could we detach the relevant tensors during the forward propagation and manually propagate the gradients during the backward propagation?</p>

soulitzer: <p>Would multithreading work instead for your case?</p>
Tuning a network, subset of data, one non-frozen layer

emerth: <p>For the sake of this post, say I am training googlenet using 500 classes from imagenet, and I have done some number of epochs and then looked at visualizations for each layer. And I have determined that I want to use one particular class from my 500 to train exclusively one layer.</p>
<p>Let’s say I want to train layer model.inception5b.branch2[0].conv using only imagenet class n04026813.</p>
<p>I froze all the other layers using this idiom:</p>
<pre><code class="lang-auto"> # layers_to_freeze contains refs to every layer except 
 # model.inception5b.branch2[0].conv and the final fc layer.
 for layer in layers_to_freeze: 
    for param in layer.parameters():
       param.requires_grad = False
</code></pre>
<p>Now, I would like to train the model using only, say, imagenet class n04026813, so as to (try to) make  the chosen unfrozen layer learn this class really well.</p>
<p>Problem I have it this:</p>
<p>If I set up a suite of directories data/train/[500 class dirs] and data/val/[500 class dirs] with all but */n04026813/ empty, then the data loader chokes on not finding any files in 499 of the class dirs.</p>
<p>If I instead stick one image in each of the 499 class dirs that I want to ignore (so that the data loader is happy) then I think I must be teaching that unfrozen layer mainly to memorize those  499 files.</p>
<p>Question: how can I get the dataloader to be happy with empty class directories? Or put it another way - how can I make pytorch just ignore the other 499 classes?</p>
<hr>
<p>IDK if my thinking here is just wrong in the context of pytorch. I’ve done this kind of thing using Caffe in the past, but in Caffe you say in the prototxt that there are X number of classes, and if (X-1) of the corresponding train &amp; val directories happen to not contain any data then that’s fine, those classes just are not represented.</p>
How to save/load PyTorch model with custom Storage pickling

brandon.pelfrey: <p>I’m looking to store PyTorch model checkpoints using the typical format, except I do not want the tensor Storage data to be inside this pickle. I want to control where the tensor data is pickled and stored. (Storage is going to take place via RDMA, avoiding system memory/zipping/etc.). I would like to provide my own pickler to the <code>torch.save()</code> method, except that <a href="https://github.com/pytorch/pytorch/blob/main/torch/serialization.py#L1206-L1208" rel="noopener nofollow ugc">doesn’t allow me to control the important part</a> (the <code>persistent_id</code> method). How can I store/load tensor Storage data using my own logic while allowing the rest to flow through the normal checkpointing logic?</p>
Problem with FSDP, custom gradient

nasosg10: <p>Hi everyone!<br>
I want to build a specific implementation with PyTorch and train it with FSDP, and I am facing some problems.<br>
First of all, let me describe my implementation with pseudocode.<br>
Let model be a nn.Module with a shared submodule (also nn.Module), and the heads being also submodules (and nn.Modules). I have implemented the following pseudocode in PyTorch, and due to memory constraints, I have to use FSDP to train it.</p>
<p>z = model.shared(x)<br>
d = z.detach()<br>
d.requires_grad = True</p>
<p>for i in range(n):<br>
&nbsp;&nbsp;&nbsp;&nbsp;p = model.heads[i](d)<br>
&nbsp;&nbsp;&nbsp;&nbsp;loss = loss_function(p, y[i])<br>
&nbsp;&nbsp;&nbsp;&nbsp;loss.backward()<br>
&nbsp;&nbsp;&nbsp;&nbsp;# gradients accumulate in d</p>
<p>z.backward(gradient=d.grad)</p>
<p>However, there are some limitations.<br>
In my original implementation, all of this code was included in the forward method of the model, but with FSDP, we can’t call backward inside the forward pass. So I have made some if-else statements in the forward, to know if I want to use the shared part or a head, and I wrote the sequential forward-backward logic in my training loop. However, when the code reaches the last line (where I want to do backwards with custom gradient), it breaks, and logs a RunTime error :</p>
<p>setStorage: sizes [4096, 14336], strides [14336, 1], storage offset 159383552, and itemsize 2 requiring a storage size of 436207616 are out of bounds for storage of size 0.</p>
<p>I suppose the error is due to the custom gradient, since the other backward passes were executed without problems. But is there any workaround to fix this?<br>
Or is there any other way to use FSDP to train a model according to this pseudocode?</p>
ERROR: Could not find a version that satisfies the requirement pytorch-triton==2.3.1+958fccea74

Krytern: <p><code>ERROR: Could not find a version that satisfies the requirement pytorch-triton==2.3.1+958fccea74 (from versions: 2.0.0+0d7e753227, 2.0.0+3aa3d7024e, 2.0.0+af76c989eb, 2.0.0+b8b470bc59, 2.0.0+c8bfe3f548, 2.0.0+d54c04abe2, 2.1.0, 2.1.0+2c32f43999, 2.1.0+3c400e7818, 2.1.0+440fd1bf20, 2.1.0+46672772b4, 2.1.0+6e4932cda8, 2.1.0+7d1a95b046, 2.1.0+9820899b38, 2.1.0+9e3e10c5ed, 2.1.0+bcad9dabe1, 2.1.0+e6216047b8, 2.1.0+e650d3708b, 2.2.0+e28a256d71, 3.0.0+45fff310c8, 3.0.0+757b6a61e7, 3.0.0+901819d2b6, 3.0.0+989adb9a29, 3.0.0+a9bc1a3647, 3.0.0+dedb7bdf33, 3.1.0+5fe38ffd73, 3.1.0+cf34004b8a, 3.2.0, 3.2.0+git0d4682f0, 3.2.0+git35c6c7c6, 3.2.0+git4b3bb1f8, 3.2.0+gitb2684bf3, 3.2.0+gitf9cdf582, 3.3.0+git96316ce5, 3.3.0+gitab727c40) ERROR: No matching distribution found for pytorch-triton==2.3.1+958fccea74</code></p>
<p>This is an error when using <code>make triton</code> command for v2.3.1 of Pytorch. I can’t find the exsistance of <code>pytorch-triton==2.3.1+958fccea7</code> anywhere on the internet. When I try a 2.3.1 version from <a href="https://download.pytorch.org/whl/test/pytorch-triton/" rel="noopener nofollow ugc">https://download.pytorch.org/whl/test/pytorch-triton/</a> it doesn’t detect it giving me the same error. I’m using Python version 3.11.11</p>
unstable WGAN-GP gradients

Szczerku: <p>I do not understand why, when I run the code with the network wgan-gp, one time my generator receives super gradients and the network is stable, I can see the improvement from the generator and the critic. and another time when I run the code it is practically from the very beginning that the generator does not receive gradients and does not learn. is it possible to make it stable? so that the network learns every time I run it and not so randomly?</p>

J_Johnson: <p>GANs are inheritently unstable.</p>
<p>But, since you’re using Wasserstein loss already(i.e. <strong>W</strong>GAN), sounds to me like the learning rate on your optimizer might be set too high. Try lowering that and see if it helps.</p>
How to set the default device to torch::kCUDA?

Amadou: <p>Hello,</p>
<p>How can I set the default device to CUDA with libtorch C++?</p>
<p>Thanks in advance!</p>

beazt: <p>You can do it like so:</p>
<pre><code class="lang-auto">torch::Device device = torch::kCPU;
if (torch::cuda::is_available()) {
  device = torch::kCUDA;
}
</code></pre>

Amadou: <p><a class="mention" href="/u/beazt">@beazt</a> thanks for your answer, but What I want to get is like:</p>
<pre><code class="lang-auto">auto tensor = torch::randn({2,2})
</code></pre>
<p>Here, the default device of tensor is kCPU <code>tensor.device()</code>. But I want for all tensors that I created switch the default device to kCUDA without doing this <code>my_tensor = my_tensor.to(torch::kCUDA)</code> for each tensor.</p>
<p>Is there a way to do this simultaneously for all created tensors ?</p>

beazt: <p>As far as I know, there is no way to do it automatically at initialization. You have to do it manually for each tensor, I’m afraid.</p>

Amadou: <p><a class="mention" href="/u/beazt">@beazt</a> Thanks a lot!</p>
<p>I thinked about fast solution. I will use the manual method. Thanks</p>

neoncube: <p>For anyone running across this in 2024, see <code>set_default_device()</code>: <a href="https://pytorch.org/docs/stable/generated/torch.set_default_device.html" class="inline-onebox" rel="noopener nofollow ugc">torch.set_default_device — PyTorch 2.6 documentation</a></p>
ExecuTorch: Variable-Length Inputs for Export and F.maxpool1d

cba913: <p>My PyTorch model takes variable-length inputs. It uses max-pooling at several locations to downsample features.</p>
<p>I would like to export this model to ExecuTorch (*.pte), to run on mobile.</p>
<p>I just cannot for the life of me get the variable-length part working (it can easily export with fixed-length inputs).</p>
<p>There are several tutorials and forums (sorry, new users can only put two links in a post):</p>
<ul>
<li><a href="https://pytorch.org/executorch/stable/tutorials/export-to-executorch-tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Exporting to ExecuTorch Tutorial — ExecuTorch 0.5 documentation</a></li>
<li><a href="https://pytorch.org/tutorials/intermediate/torch_export_tutorial.html#constraints-dynamic-shapes" class="inline-onebox" rel="noopener nofollow ugc">torch.export Tutorial — PyTorch Tutorials 2.6.0+cu124 documentation</a></li>
</ul>
<p>None of the presented syntaxes solved my issue.</p>
<p>I am using PyTorch version ‘2.6.0+cu124’.</p>
<p>Fixed Length:</p>
<blockquote>
<p>dummy_dim = 20000<br>
sample_args = (torch.randn(1,1, dummy_dim),)<br>
ex = torch.export.export(model, sample_args)<br>
test_inp = torch.randn(1, 1, dummy_dim)<br>
ex.module()(test_inp) # → returns model output</p>
</blockquote>
<p>Variable Length (this is the only code I managed to export):</p>
<blockquote>
<p>dummy_dim = 20000<br>
sample_args = (torch.randn(1,1, dummy_dim),)<br>
dynamic_shapes = {“x”: {2: torch.export.Dim.AUTO}}<br>
ex = torch.export.export(model, sample_args, dynamic_shapes=dynamic_shapes)<br>
test_inp1 = torch.randn(1, 1, dummy_dim)<br>
test_inp2 = torch.randn(1, 1, dummy_dim // 2)<br>
ex.module()(test_inp1) # → returns model output, same shape as dummy<br>
ex.module()(test_inp2) # → RuntimeError: Expected input at *args[0].shape[2] to be equal to 20000, but got 10000</p>
</blockquote>
<p>Upon further inspection of the logs, I found the following:</p>
<p>(1) I0405 19:44:30.139000 67963 site-packages/torch/fx/experimental/symbolic_shapes.py:4423] [0/0] create_symbol s0 = 20000 for L[‘x’].size()[2] [5000, 300000] (_dynamo/variables/builder.py:2861 in ), for more info run with TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL=“s0” or to suppress this message run with TORCHDYNAMO_EXTENDED_ADVICE=“0”<br>
(2) I0405 19:44:30.256000 67963 site-packages/torch/fx/experimental/symbolic_shapes.py:6281] [0/0] eval Eq(s0 - 250, 19750) [guard added] x = F.max_pool1d(x, self.maxpool)  # model.py in forward (nn/functional.py:740 in _max_pool1d)</p>
<p>What I think happens is, a guard is added on the maxpool because the symbolic placeholder cannot verify the shape (it’s dynamic at runtime in the variable-length model in PyTorch). From the output, we can see that the size of the final dimension is always fixed to the length of the dummy input.</p>
<p>I am just not sure of the specific syntax to resolve this error. Can anybody help?</p>
How to custom a quantizar using fx

BambooKui: <p>I want to know how to implement some quantization algorithms using PyTorch’s FX.</p>

jerryzh168: <p>please give more details on what quant algorithms you want to implement, here is the official doc for fx: <a href="https://pytorch.org/docs/stable/fx.html" class="inline-onebox" rel="noopener nofollow ugc">torch.fx — PyTorch 2.6 documentation</a></p>
Quantized LLM inference vs quantized matrix multiplication speed in CPU

luentong: <p>Hi!<br>
I did some experiments on quantized LLM inference vs quantized matrix multiplication speed in CPU, for both x86 Ubuntu machine and Kunpeng EulerOS machine.</p>
<p>Matrix muliplication speed:<br>
x86 Ubuntu Python 1000x1000 bfloat16 0.87-0.88 sec<br>
x86 Ubuntu Python 1000x1000 float16 0.83-0.84 sec<br>
x86 Ubuntu Python 1000x1000 float32 0.015-0.03 sec<br>
arch64 Euler Python  1000x1000 float32 0.021-0.022 sec<br>
arch64 Euler Python  1000x1000 float16 0.062-0.064 sec<br>
arch64 Euler Python 1000x1000 bfloat16 0.043-0.045 sec</p>
<p>Qwen2.5-0.5B-Instruct:<br>
x86 Ubuntu CPU inference，bfloat16 :11.21 sec<br>
x86 Ubuntu CPU inference，float16 :2.70 sec<br>
x86 Ubuntu CPU inference，float32 :24.84 sec<br>
arch64 Euler CPU inference，bfloat16: 26.96 sec<br>
arch64 Euler CPU inference，float16: 11.62 sec<br>
arch64 Euler CPU inference，float32: 27.2 sec</p>
<p>LLM inference speed:</p>
<p>Video-Llama-3-2b:<br>
x86 Ubuntu CPU， bfloat16：11.73 sec<br>
x86 Ubuntu CPU ， float16：23.00 sec<br>
x86 Ubuntu CPU， float32：140.00 sec<br>
arch64 Euler CPU， bfloat16：333.67 sec<br>
arch64 Euler CPU， float16：49.55 sec<br>
arch64 Euler CPU， float32：189 sec</p>
<p>So for some LLMs, it’s running faster with float16/bfloat16 than float32 on CPU, as expected. But for pytorch matrix multiplication, it always doesn’t. I’m wondering how this works? Basically how does certain LLM implement speed-ups on quantized calculations that goes over what Pytorch can do.</p>

jerryzh168: <p>looks like this is not quantized.</p>
<p>maybe you can try out our new int4 cpu from intel:</p>
<pre><code class="lang-auto">pip install torchao
</code></pre>
<pre><code class="lang-auto">from torchao import quantize_
from torchao.quantization import Int4WeightOnlyConfig
from torchao.dtypes import Int4CPULayout

quantize_(model, Int4WeightOnlyConfig(group_size=128, layout=Int4CPULayout())

</code></pre>

jerryzh168: <p>in the end these will run the quantized ops, such as for int4 weight only cuda gpu, we have: <a href="https://github.com/pytorch/pytorch/blob/cfea55dbecf93a88a40290a69c5e3b324dcec69c/aten/src/ATen/native/native_functions.yaml#L4163" class="inline-onebox" rel="noopener nofollow ugc">pytorch/aten/src/ATen/native/native_functions.yaml at cfea55dbecf93a88a40290a69c5e3b324dcec69c · pytorch/pytorch · GitHub</a></p>
<p>for bfloat16/float16 etc. they just use the same op, and there will be some dispatch inside the op based on dtype that routes the compute to different kernels for each dtype</p>
What is the common base class for my module (holders) in libtorch?

KiwiPower: <p>Consider the following snippet of C++:</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">
struct Simple1Impl : public torch::nn::Module
{
    /* Constructor and forward */
};
TORCH_MODULE( Simple1 );

struct Simple2Impl : public torch::nn::Module
{
    /* Constructor and forward */
};
TORCH_MODULE( Simple2 );
</code></pre>
<p>What is the common base class for which I can write things such as:</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">WhatIsThisType model = condition ? (SomeCast) Simple1() : (SomeCast) Simple2();
</code></pre>
<p>I would like to dynamically decide the model. I have tried <code>std::shared_ptr&lt;torch::nn::Module&gt;</code> and <code>torch::nn::AnyModule</code>, both of which cause compilation errors.</p>
<p>As a reminder, the macro definition is this:</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">/// Defines a class `Name` which inherits from `nn::ModuleHolder` to provide a
/// wrapper over a `std::shared_ptr&lt;ImplType&gt;`.
/// `Impl` is a type alias for `ImplType` which provides a way to call static
/// method of `ImplType`.
#define TORCH_MODULE_IMPL(Name, ImplType)                              \
  class Name : public torch::nn::ModuleHolder&lt;ImplType&gt; { /* NOLINT */ \
   public:                                                             \
    using torch::nn::ModuleHolder&lt;ImplType&gt;::ModuleHolder;             \
    using Impl TORCH_UNUSED_EXCEPT_CUDA = ImplType;                    \
  }

/// Like `TORCH_MODULE_IMPL`, but defaults the `ImplType` name to `&lt;Name&gt;Impl`.
#define TORCH_MODULE(Name) TORCH_MODULE_IMPL(Name, Name##Impl)
</code></pre>
<p>That would make <code>Simple1</code> a class derived from <code>public::nn::ModuleHolder&lt;Simple1Impl&gt;</code>. If the latter class behaves like <code>std::shared_ptr&lt;Simple1Impl&gt;</code>, then one should conceivably be able to cast the latter <code>public::nn::ModuleHolder&lt;torch::nn::Module&gt;</code>, though that may depend on the casting operators.</p>

KiwiPower: <p>For the <code>std::shared_ptr</code> class, the construction via derived pointers seems to work because the constructor is templated:<br>
<a href="https://en.cppreference.com/w/cpp/memory/shared_ptr/shared_ptr" rel="noopener nofollow ugc">https://en.cppreference.com/w/cpp/memory/shared_ptr/shared_ptr</a></p>
<p>However, that behavior is not provided by the wrapper class <code>ModuleHolder</code>, which seems to only accept assignments or copy/move constructions with one specific class.</p>
<p>So that possible seems out, for now. I don’t know whether that omission is by deliberate design.</p>

KiwiPower: <p>I am sharing a makeshift dont-know-better solution for anyone who happens to wonder about the same thing.</p>
<p>It seems that a one-item <code>torch::nn::Sequential</code> can serve as a universal container for modules. Hence you might write:</p>
<pre><code class="lang-auto">torch::nn::Sequential model = condition ? torch::nn::Sequential( Simple1() ):  torch::nn::Sequential( Simple2() );
</code></pre>
AMP mixed precision in custome module: RuntimeError: hook '<lambda>' has changed the type of value

pytorcher: <p>I have a module which uses backwards hooks, buffers, and parameters, that works properly with either float of half tensors when specified.  I am now using it with AMP mixed precision for the first time and it is causing this error, but I don’t know what to do to resolve it.  I have set <code>torch.autograd.set_detect_anomaly(True)</code> which grants the following error:</p>
<pre><code class="lang-auto">ENVdavit/lib/python3.12/site-packages/torch/autograd/graph.py:823: UserWarning: 
Error detected in SigmoidBackward0. Traceback of forward call that caused the error:
  File
......
  File "myModule.py", line 878, in forward
    Outs[i] = torch.sigmoid(earlierOuts[i]).to(device)
 (Triggered internally at /pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:122.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File line 827, in train_one_epoch
    loss_scaler(
  File "davit/timm/utils/cuda.py", line 43, in __call__
    self._scaler.scale(loss).backward(create_graph=create_graph)
  File "ENVdavit/lib/python3.12/site-packages/torch/_tensor.py", line 626, in backward
    torch.autograd.backward(
  File "ENVdavit/lib/python3.12/site-packages/torch/autograd/__init__.py", line 347, in backward
    _engine_run_backward(
  File "ENVdavit/lib/python3.12/site-packages/torch/autograd/graph.py", line 823, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
           ^^^^^^^^^^^

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: hook '&lt;lambda&gt;' has changed the type of value (was torch.cuda.HalfTensor got torch.cuda.FloatTensor)
</code></pre>