akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Cuda out of memory when restart from checkpoint

Chris_YU: <p>I know this is a pretty old topic. I am using model parallel with DDP: the model is partitioned into four parts, each residing on a GPU. When I restart training from a checkpoint, it gives Cuda out of memory error on GPU1 (largest model partition). I am using AdamW as optimizer and gradient checkpointing. The loading process looks like below:</p>
<pre><code class="lang-auto">checkpoint = torch.load('checkpoint.pth', map_location='cpu')
model = MyModel()
model.load_state_dict(checkpoint['model'])
optimizer.load_state_dict(checkpoint['optimizer'])

model.to_device(device_list) #distribute model to 4 GPU
model = DDP(model, device_ids=None, output_device=None)

del checkpoint
torch.cuda.empty_cache()
</code></pre>
<p>Tried different methods online but nothing worked.</p>
Reasons behind using convolutional layers of 1x1 kernel sizes

amirtmgr: <p>I came across this paper called, <a href="https://www.researchgate.net/publication/350757416_Boosting_Inertial-Based_Human_Activity_Recognition_With_Transformers" rel="noopener nofollow ugc">Inertial-based Activity Recognition with Transformers</a>, where authors have utilized CNN + Transformer encoders + FCs to classify human activities from sensor data.</p>
<p>I deep-dived into the code implementation and found they are using Convolutional layers of kernel size 1x1. Can anyone explain why the author chose 1x1 kernel-size convolutional layers? What are the significance or effects of these in the model?</p>
<p>To my understanding, the Conv layers with 1x1 kernel size generate new high-level features incorporating data from all channels without changing the step sizes.</p>
<p>Am I right here? Can someone explain the significance or effects of using 1x1 kernel size in such a scenario of human activity recognition from sensor data?</p>

anantguptadbl: <p>Sounds correct to me. Best way is to visualize the output from a single image.<br>
You can try this in a jupyter notebook</p>
<pre><code class="lang-auto">from sklearn.datasets import load_sample_image
import matplotlib.pyplot as plt
import numpy as np

%matplotlib inline
flower = load_sample_image('flower.jpg') 
print(flower.shape)
plt.imshow(flower)
plt.figure()

flower_torch = torch.from_numpy(flower.transpose(2,0,1).astype(np.float32))

conv_layer = nn.Conv2d(in_channels=3, out_channels=3, kernel_size=1, stride=1, padding=0)
output = conv_layer(flower_torch)
print(output.detach().numpy().shape)
plt.imshow(output.detach().numpy().transpose(1,2,0))
</code></pre>
CUDA Graphs with Torch Library API

MatanT: <p>Hi I have an implementation of a kernel in CUDA C++.<br>
I am using it with torch library API in my python code.<br>
Everything works great (the kernel runs on the GPU on regular calls), but when I try to capture a graph with the code, I get a message that the graph is empty (and replay does not runs the kernel of course).<br>
I would apricate any help to know why this is happening, and any advice on how to capture a kernel in PyTorch which is implemented in CUDA C++.<br>
Thanks.</p>

ptrblck: <p>I remember seeing a similar issue recently here where a user forgot to pass the surrounding CUDAStream to the custom kernel, but I cannot find the thread right now. Could this be the case here, too?</p>

MatanT: <p>Wow thanks!!! this works now. Why is that the case though?</p>

ptrblck: <p>From the <a href="https://pytorch.org/docs/stable/notes/cuda.html#constraints">docs</a>:</p>
<blockquote>
<p>Capture must occur on a non-default stream</p>
</blockquote>
<p>which will be set as the default stream in the context manager and has to be passed to custom functions as well.</p>
UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it

vince62s: <p>Hi,</p>
<p>I am getting this warning and would like to track down which operator triggers this:</p>
<p>lib/python3.11/site-packages/torch/autograd/graph.py:744: UserWarning: c10d::allreduce_: an autograd kernel was not registered to the Autograd key(s) but we are trying to backprop through it. This may lead to silently incorrect behavior. This behavior is deprecated and will be removed in a future version of PyTorch. If your operator is differentiable, please ensure you have registered an autograd kernel to the correct Autograd key (e.g. DispatchKey::Autograd, DispatchKey::CompositeImplicitAutograd). If your operator is not differentiable, or to squash this warning and use the previous behavior, please register torch::CppFunction::makeFallthrough() to DispatchKey::Autograd. (Triggered internally at /opt/conda/conda-bld/pytorch_1716905969073/work/torch/csrc/autograd/autograd_not_implemented_fallback.cpp:63.)<br>
return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass</p>
<p>Thanks.</p>

vince62s: <p>does this simply mean that the allreduce_ from c10d is not Autograd / backward compatible ?<br>
if so how can we perform this ?</p>

tbenthompson: <p>I’m hitting this same warning. Any info would be useful! Thanks!</p>

karajan: <p>I believe this is because it’s trying to backpropogate through <code>torch.distributed.all_reduce</code>, which is not possible at the moment.</p>
<p>Instead, you can replace <code>torch.distributed.all_reduce</code> with <code>torch.distributed.nn.functional.all_reduce</code>, which has the <code>backward</code> method implemented. See the suggestion at for more details and caveats: <a href="https://discuss.pytorch.org/t/do-gradients-propagate-through-all-reduce-all-gather/159572/2" class="inline-onebox">Do gradients propagate through all_reduce &amp; all_gather? - #2 by wanchaol</a></p>
<p>Also refer to <a href="https://github.com/pytorch/pytorch/issues/58005#issuecomment-1778029156" class="inline-onebox" rel="noopener nofollow ugc">torch.distributed.nn.all_reduce incorrectly scales the gradient · Issue #58005 · pytorch/pytorch · GitHub</a> for caveats around the (in-)correctness of the gradient if reduce ops is not sum with <code>torch.distributed.nn.functional.all_reduce</code>.</p>
For a model takes custom type as input, can I customize scatter function for DataParallel?

liabrarristsh: <p><code>Mymodel</code> has too much inputs so I’ve created a class <code>MyInput</code> to handle it. The class has many list of tensors as property, and each tensor in the lists is a (batched) input.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch as pt

class MyInput: # batched input
    device: pt.device # device of all the Tensors below
    x: list[pt.FloatTensor]
    mask: list[pt.BoolTensor]
    idx: list[pt.IntTensor]
    # each item in the list is one of the inputs of MyModel, 
    # with batch at first dim
    
    def __init__(self, 
                x: list[pt.FloatTensor], 
                mask: list[pt.BoolTensor], 
                idx: list[pt.IntTensor]
            ) -&gt; None:
        self.x, self.mask, self.idx = x, mask, idx
    # other things of MyInput
    ...

class MyModel(pt.nn.Module): 
    # Implementation of MyModel
    ...

    def forward(self, batch: MyInput) -&gt; pt.FloatTensor: 
        # forward function with batch as input, e.g.
        a = self.a_certain_layer(batch.x[3])
        b = a[batch.mask[2]]
        c = b[:, batch.idx[3]]
        return c
</code></pre>
<p>Obviously, <code>Dataparallel</code> cannot process <code>MyInput</code> by default, so I plan to write a custom scatter function for <code>MyInput</code>, like this:</p>
<pre data-code-wrap="python"><code class="lang-python">def scatter_MyInput(all_batch: MyInput) -&gt; list[MyInput]: 
    x_s = default_scatter(all_batch.x)
    # default_scatter is the default scatter function of DataParallel
    m_s = default_scatter(all_batch.mask)
    i_s = default_scatter(all_batch.idx)
    return [MyInput(*props) for props in zip(x_s, m_s, i_s)]
</code></pre>
<p>So,</p>
<ol>
<li>How can I tell <code>DataParallel</code> to use custom <code>scatter_MyInput</code> instead of default scatter function?</li>
<li>Where can I import <code>default_scatter</code>?</li>
<li>Does <code>default_scatter</code> work for listed batched tensor? Or do I have to scatter the item in list with <code>default_scatter</code> one by one?</li>
</ol>
Sm_86 not supported by nightly build

LDec-AlmostHuman: <p>I’m getting warning that my card which is sm_86 is not supported by nightly build. Are you removing all cards under sm_90 from support in new PyTorch?</p>

ptrblck: <p>No, sm_86 is still supported in all of our nightly binaries and we are not planning to remove it.</p>

LDec-AlmostHuman: <p>OK, please look at below message.</p>
<p>Nightly build CUDA 12.6.3 and CUDA 12.8.0 message:</p>
<p>python3 -c “import torch; print(torch.cuda.is_available()); print(torch.cuda.device_count()); print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else ‘No CUDA’)”<br>
True<br>
1<br>
/usr/local/lib/python3.12/dist-packages/torch/cuda/<strong>init</strong>.py:266: UserWarning:<br>
Found GPU0 NVIDIA RTX A4000 which is of cuda capability 8.6.<br>
PyTorch no longer supports this GPU because it is too old.<br>
The minimum cuda capability supported by this library is 9.0.</p>
<p>warnings.warn(<br>
/usr/local/lib/python3.12/dist-packages/torch/cuda/<strong>init</strong>.py:291: UserWarning:<br>
NVIDIA RTX A4000 with CUDA capability sm_86 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_90.<br>
If you want to use the NVIDIA RTX A4000 GPU with PyTorch, please check the instructions at <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>
<p>warnings.warn(<br>
NVIDIA RTX A4000</p>

ptrblck: <p>Are you using an ARM system? In this case it’s expected as we support Hopper and Blackwell. x86 binaries support Maxwell to Hopper using CUDA 12.6 and Turing to Blackwell using CUDA 12.8.</p>

LDec-AlmostHuman: <p>Yes, but not NVidia specific HW. I’m on AsrockRack ARM64 Neoverse-N1 motherboard.<br>
Actually that is sad about ARM64 that you have removed GPUs like L40s, RTX 6000, etc.<br>
RTX 6000 Blackwell would be my upgrade choice I believe, but this, I think, it’s expected in Q3 2025.</p>
<p>Pozdrawiam<br>
Łukasz</p>
Does slicing/trimming during training cause memory leak?

s63jgn: <p>During training, at some point I need my tensors to be padded to the same size.<br>
Later on (still during training), I want to remove the padding by slicing and keeping only a certain part of the tensors.</p>
<p>I appears to me that this causes a cuda memory leak, since the subgraph of the removed part is not involved in the loss computation/backprop then and thus it is never cleaned up. Is my understanding correct about this?</p>
<p>And if so, how can I “trim” a tensor during training without running into this memory leak?</p>

ptrblck: <p>No, PyTorch will not leak memory, which would mean you cannot deallocate the data and have lost it. The intermediate tensors would of course still use the additional memory which will be removed later but all intermediates will be freed after the backward pass.</p>
Dockerfile to create image with python 3.13

NourEldin-Osama: <p>i want the Dockerfile that used to create the following docker image<br>
pytorch:2.6.0-cuda12.6-cudnn9-runtime<br>
i couldn’t find it on Github<br>
i want the file to create a new image but with python 3.13<br>
any help is appreciated<br>
thanks in advance</p>

ptrblck: <p>I would guess the <code>Dockerfile</code> from the <a href="https://github.com/pytorch/pytorch/blob/861bf892fb91d2301986b881bd2ef7b01a98a1dc/Dockerfile">repository</a> is used which sets up an Ubuntu base container and <code>pip install</code>s the wheels.</p>

NourEldin-Osama: <p>Thanks for your response<br>
I would love to get the final file without configurations that in the end will generate<br>
<strong>pytorch:2.6.0-cuda12.6-cudnn9-runtime</strong> but with python <strong>3.13</strong><br>
the current docker image is using python 3.11</p>
<p>Thanks in advance</p>
RAM speed on Windows 11

Erislynch: <p>Good morning, how are you all, this is my first post on this forum, I’ve been having a lot of problems with RAM speed on Windows 11 for the past few days. I don’t understand why this is happening. I’ve tried a lot, but I can’t fix it, can anyone help me with this?</p>

ptrblck: <p>Could you describe what kind of issues you are seeing exactly?</p>
Deep Conv1d NN & DataLoader: Must batch_size = in_channels of first Conv1d layer?

Geremia: <p>I’m training a NN with <code>Conv1d</code> layers using DataLoader. Must <code>data_loader</code>’s <code>batch_size</code> =  # of <code>in_channels</code> of first Conv1d layer? In my case, if it isn’t, I get matrix size mismatch errors.</p>
<p>Also, are there any tutorials on how to design convolutional NNs in PyTorch. I’m having trouble getting the <code>in_channel</code>s and <code>out_channel</code>s to be the correct sizes.</p>

ptrblck: <p>No, the batch size does not need to match the number of input channels of the first convolution as the <code>nn.Conv1d</code> layer expects input in the shape <code>[batch_size, channels, seq_len]</code>.</p>
<aside class="quote no-group" data-username="Geremia" data-post="1" data-topic="217011">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/geremia/48/70165_2.png" class="avatar"> Geremia:</div>
<blockquote>
<p>In my case, if it isn’t, I get matrix size mismatch errors.</p>
</blockquote>
</aside>
<p>If the shape mismatch is raised on the first linear layer, make sure the incoming input activation has the expected number of features (after fattening).</p>
Function selection for upsampling 1.5 times in gan

2209262843: <p>Generally, the video resolution ranges from 720p (image 1280 x 720) to 1080p (image 1920 x 1080), and upsampling is 1.5 times. However, the pixel shuffle is used for upsampling on a common gan network, which does not support non-integer multiples. Are there other functions for upsampling, and which method has better performance and precision?</p>

Mohammad_Hassan_Soha: <p>The best solution for upscaling images is to use a resize function that leverages exponential functions. Essentially, you fit an exponential curve to the data and then resample from the interpolated points on that curve to reconstruct the original signal at the higher resolution. Is there any reason you would avoid this approach?</p>
Why is it not necessary to set requires_grad=True in a class subclassing nn.Module

make1234: <p>Hello,</p>
<p>I am experimenting with pytorch and noticed that setting requires_grad=True for layers in a class subclassing nn.Module (like this <a href="https://pytorch.org/tutorials/beginner/basics/buildmodel_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Build the Neural Network — PyTorch Tutorials 2.6.0+cu124 documentation</a>) is not necessary whereas here it is. <a href="https://pytorch.org/tutorials/beginner/basics/autogradqs_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Automatic Differentiation with torch.autograd — PyTorch Tutorials 2.6.0+cu124 documentation</a><br>
Why is there this difference?</p>
<p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>When you are creating a model via <code>nn.Module</code>s, all modules with trainable parameters will initialize <code>nn.Parameter</code>s internally, which require gradients by design.<br>
You can check these via <code>print(dict(model.named_parameters())</code>. This approach is used in the first example.</p>
<p>However, you don’t necessarily need to use modules and can also write your code in a pure functional way. In this case you could create trainable parameters by creating tensors with <code>requires_grad=True</code> yourself.</p>
Python saving audio streamed to node input as speedup and pitchshifted .wav

qraqatit: <p>I am trying to create a ComfyUI node that would take image sequence and audio from respective node inputs (“IMAGE”, “AUDIO”) and saving it to an uncompressed AVI, but for some reason it is speeding-up and pitch shifting the audio.</p>
<p>This is the relevant code that I use for saving temporary wav file:</p>
<pre><code class="lang-auto">a_waveform = audio['waveform']    
temp_audio_path = self.get_unique_filename(output_dir, filename_prefix, "wav")
waveform_data = a_waveform.cpu().numpy()
sample_rate = audio['sample_rate']
write(temp_audio_path, sample_rate, waveform_data) 
</code></pre>
torch.OutOfMemoryError: CUDA out of memory when running xtremedistil-l12-h384-uncased model

chrathans: <p>I am loading a model in my GPU and try to check how it performs when generating text. More specifically, I have the following code:</p>
<pre><code class="lang-auto">model_name = "microsoft/xtremedistil-l12-h384-uncased"  # Find popular HuggingFace models here: https://huggingface.co/models

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map = "cuda",
    torch_dtype = "auto",
    trust_remote_code = True,
    output_attentions=True
)

# Create a pipeline
generator = pipeline(
    "text-generation",
    model = model,
    tokenizer = tokenizer,
    return_full_text= False,
    max_new_tokens = 500,
    do_sample = False
)

input_text = "What is the co-capital of Turkey according to citizens opinions: "  
inputs = tokenizer.encode(input_text, return_tensors='pt')  # Tokenize input text
input_ids = tokenizer(input_text, return_tensors = "pt").input_ids

# tokenize the input prompt
input_ids = input_ids.to("cuda")

output = generator(messages)
</code></pre>
<p>However, when trying to run this code I receive the following error:</p>
<pre><code class="lang-auto">    attention_scores = attention_scores / math.sqrt(self.attention_head_size)
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 12.00 MiB. GPU 0 has a total capacity of 23.57 GiB of which 5.88 MiB is free. Including non-PyTorch memory, this process has 23.55 GiB memory in use. Of the allocated memory 20.55 GiB is allocated by PyTorch, and 2.71 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
</code></pre>
<p>What is this message is telling me? Can I overcome this issue?</p>
Slow attention when using kvCache

ywha0929: <p>I am currently using rtx4060 for desktop, and Jetson AGX Orin board for research.<br>
When I make inference using Llama3.2 1B model, whether I am using kvCache(past_key_values) changes the attention implementation (backend)<br>
In rtx4060 setup, using kvCache reduced the inference time, but in Jetson board, using kvCache increased the inference time.<br>
According to pytorch profiler, I found that Jetson’s choice for attention when using kvCache was scaled_dot_product_cudnn_attention. And this takes extremely long time on first iteration (over 100times compared to without cache version)<br>
Any one knows why this happended and how I can fix this?</p>
<p>The below image is a snapshot of profiler<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/48b8b2fe5bbc5b043e270063d7b0fd56f6ac7909.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/48b8b2fe5bbc5b043e270063d7b0fd56f6ac7909" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48b8b2fe5bbc5b043e270063d7b0fd56f6ac7909_2_690x201.jpeg" alt="image" data-base62-sha1="ank5PBt8woE81f1s4rowY3CVRkB" width="690" height="201" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48b8b2fe5bbc5b043e270063d7b0fd56f6ac7909_2_690x201.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48b8b2fe5bbc5b043e270063d7b0fd56f6ac7909_2_1035x301.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/48b8b2fe5bbc5b043e270063d7b0fd56f6ac7909_2_1380x402.jpeg 2x" data-dominant-color="E2D6DF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1667×487 160 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Assuming the slow iteration time is only visible in tbe first iteration, you would see the runtime compilation overhead.</p>
Adding/removing new trainers on the single node to elastic training

wl2776: <p>Documentation for <code>torchrun</code> says about parameter <code>--nnodes=1:4</code>, that allows to run training on one, two, three or four PCs, each of which has <code>$NUM_TRAINERS</code> GPU.</p>
<p>Is it possible to add or remove new trainers to a group on a single node?</p>
<p>My workstation has two GPUs, and I am using <a href="https://github.com/justanhduc/task-spooler" rel="noopener nofollow ugc">GPU-capable fork of task-spooler</a> for job queue. I’d like to use one GPU for development during working days and reuse it for processing over nights.</p>
torch.optim.SGD results in NaN

hiepnguyen034: <p>I followed this <a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#autograd" rel="nofollow noopener">tutorial</a> and tried to modify it a little bit to see if I understand things correctly. However, when I tried to use torch.opim.SGD</p>
<pre><code class="lang-auto">import torch
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
device = torch.device("cuda:0")

x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)
w1 = torch.nn.Parameter(torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True))
w2 = torch.nn.Parameter(torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True))
lr = 1e-6
optimizer=torch.optim.SGD([w1,w2],lr=lr)
for t in range(500):
    layer_1 = x.matmul(w1)
    layer_1 = F.relu(layer_1)
    y_pred = layer_1.matmul(w2)
    loss = (y_pred - y).pow(2).sum()
    print(t,loss.item())
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
</code></pre>
<p>My loss blows up to Inf at the third iteration and to nan afterwards, which is completely different compared to updating it manually. The code for updating it manually is below (also in the tutorial link).</p>
<pre><code class="lang-auto">x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)


w1 = torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True)
w2 = torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True)

learning_rate = 1e-6
for t in range(500):
    y_pred = x.mm(w1).clamp(min=0).mm(w2)
    loss = (y_pred - y).pow(2).sum()
    print(t, loss.item())
    loss.backward()

    with torch.no_grad():
        w1 -= learning_rate * w1.grad
        w2 -= learning_rate * w2.grad

        w1.grad.zero_()
        w2.grad.zero_()
</code></pre>
<p>I wonder what is wrong with my modified version. When I replaced SGD with Adam, the results came out pretty nice (decreasing after each iteration, no Inf or nan).</p>

Geremia: <p>I get this issue, too, where SGD → NaN, but optimizers like Adam work.</p>
<p>Try initializing the weights to something else, like 1:</p>
<pre><code class="lang-auto">w1.data.fill_(1)
w2.data.fill_(1)
</code></pre>
<p>Or a Gaussian (mean = 0, std = 1):</p>
<pre><code class="lang-auto">w1.data = torch.randn(w1.shape)
w2.data = torch.randn(w2.shape)
</code></pre>
Converting External CUDA Streams to torch.cuda.Stream()

HyperHyper: <p>Hello. I have a question regarding PyTorch Stream.</p>
<p>If I create a CuPy stream, is it possible to convert it to a <code>torch.cuda.Stream()</code>?<br>
I’m also interested in whether the same can be done for an RMM stream that I created. I’ve tried importing the RMM stream into PyTorch, but I haven’t been able to convert it into a <code>torch.cuda.Stream()</code>.</p>
<p>RMM Stream: <a href="https://github.com/rapidsai/rmm/blob/branch-25.04/python/rmm/rmm/pylibrmm/stream.pyx" class="inline-onebox" rel="noopener nofollow ugc">rmm/python/rmm/rmm/pylibrmm/stream.pyx at branch-25.04 · rapidsai/rmm · GitHub</a></p>
<hr>
<p>I found this but still not matched: <a href="https://discuss.pytorch.org/t/is-torch-cuda-stream-cuda-stream-equivalent-to-cudastream-t/130696" class="inline-onebox">Is torch.cuda.Stream().cuda_stream equivalent to cudaStream_t</a></p>
Two models in distributed data parallel

mf2055: <p>Hi, I have two models with the following characteristic and I am trying to implement DPP.</p>
<p>Model 1: a CNN type network<br>
Model 2: input is the encoder feature maps from Model 1</p>
<p>The gradient for the two models are calculated using the overall loss loss, and back propagated using their respective optimisers.</p>
<p>Overall Loss = Loss 1 + Loss 2<br>
Loss 1: Model 1’s Cross entropy and dice loss<br>
Loss 2:  Calculated by comparing Model 2 prediction and Loss 1 output</p>
<p>In this case, I want both of the model to be on the same GPU during training, since they are somewhat dependent on each other.</p>
<ol>
<li>Can we achieve this by creating the DDP environment once, and then wrapping the model using the DDP wrapper and specifying the same local rank for both network? (code below)</li>
</ol>
<p>os.environ[‘MASTER_ADDR’] = ‘localhost’<br>
os.environ[‘MASTER_PORT’] = port # str(port)<br>
torch.cuda.set_device(torch.device(‘cuda’, dist.get_rank()))<br>
dist.init_process_group(“nccl”, rank=rank, world_size=world_size)</p>
<p>self.Model 1= DDP(self.Model 1, device_ids=[self.local_rank])<br>
self.Model 2=DDP(self.Model 2,device_ids=[self.local_rank])</p>
<p>Please advise me on this!</p>
<p>Thank you!</p>
How to install Torch version that supports RTX 5090 on Windows? - CUDA kernel errors might be asynchronously reported at some other API call

FurkanGozukara: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a> . There are mentions of nightly releases but i can’t find for Windows</p>
<p>I hope you can help</p>
<p>Here my issue thread : <a href="https://github.com/pytorch/pytorch/issues/146977" class="inline-onebox" rel="noopener nofollow ugc">How to install Torch version that supports RTX 5090 on Windows? - CUDA kernel errors might be asynchronously reported at some other API call · Issue #146977 · pytorch/pytorch · GitHub</a></p>
<p>thank you</p>

ptrblck: <aside class="quote" data-post="11" data-topic="216099">
  <div class="title">
    <div class="quote-controls"></div>
    <img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/pytorch-support-for-sm120/216099/11">Pytorch support for sm120</a> <a class="badge-category__wrapper " href="/c/deployment/14"><span data-category-id="14" style="--category-badge-color: #BF1E2E; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="A category of posts focused on production usage of PyTorch. Mobile deployment is out of scope for this category (for now…  )"><span class="badge-category__name">deployment</span></span></a>
  </div>
  <blockquote>
    PyTorch binaries using CUDA 12.8 are already available as nightly binaries for Linux (x86 and SBSA). If you don’t want to use WSL and are looking for native Windows support you could check when the binaries show up <a href="https://download.pytorch.org/whl/nightly/torch/">here</a> but I will also update this thread once they are available.
  </blockquote>
</aside>


FurkanGozukara: <p>Thank you so much. Yes I don’t want WSL i need native Windows. Looking forward to that.</p>

pzlong: <p>Yes, I also urgently need native Pytorch that supports CU128. Please keep going, looking forward to it.</p>
Pytorch LSTM-VAE not able to learn

remy: <p>Hello,</p>
<p>I’m working on LSTM-VAE to make anomaly dectection on timeseries (with no constant duration). I made the code using the dropout parameter included in pytorch LSTM module.<br>
When I activate the dropout during the train phase, I have a pretty good reconstruction of the timeserie for the train phase but not for the eval phase : I only have a horizontal line for the reconstructed timeserie.<br>
When I fix dropout=0.0 (LSTM module parameter), I have horizontal line too during the train phase.<br>
Do you know where come from this behavior ? Is-it possible that the train phase is totally catched by the dropout ? If yes, what ist the solution ?<br>
Thanks by advance.</p>
Ways to pass compilation target device to dynamo backend

FatemeH: <p>Is there a way other than checking the torch.compile() input’s device type to pass the compilation target to the backend?</p>
<p>The dynamo backend I am working on can operate on multiple targets and I need to somehow differentiate between them. If all the device types were supported by pytorch, then the code would look like this:</p>
<pre><code class="lang-auto">device_type = device_from_inputs(example_inputs).type

if device_type == CPU:
   # compile for CPU
if device_type == CUDA:
   # compile for CUDA
elif device_type == D1:
   # compile for D1
elif device_type == D2:
   # compile for D2
</code></pre>
<p>The issue is that while the backend can  compile pytorch models for those devices and run the compiled model on them, some of those devices are not supported by pytorch as ATEN devices (e.g. D1 and D2).  So, I cannot create a tensor with those device types to be passed to torch.compile() . I want to know if the only solution is to implement the support for those target devices in pytorch to do that differentiation?</p>

bdhirsh: <p>I’m not sure if this completely answers your question, but: there’s a way to create custom devices out-of-tree. Check out these examples in core:</p>
<p>python code: <a href="https://github.com/pytorch/pytorch/blob/e79d9b993890dfa09d1c34c88373a23d0babd121/test/test_cpp_extensions_open_device_registration.py#L93" class="inline-onebox" rel="noopener nofollow ugc">pytorch/test_cpp_extensions_open_device_registration.py at e79d9b993890dfa09d1c34c88373a23d0babd121 · pytorch/pytorch · GitHub</a></p>
<p>cpp code: <a href="https://github.com/pytorch/pytorch/blob/main/test/cpp_extensions/open_registration_extension.cpp" class="inline-onebox" rel="noopener nofollow ugc">pytorch/open_registration_extension.cpp at main · pytorch/pytorch · GitHub</a></p>
<p>open device registration for eager mode is under active development, and we’d like to have feature parity with existing in-tree devices, so feel free to file a github issue if you think there’s any missing functionality!</p>

Tiru_B: <p><a class="mention" href="/u/bdhirsh">@bdhirsh</a>  thank you for your repo <a href="https://github.com/bdhirsh/pytorch_open_registration_example/" class="inline-onebox" rel="noopener nofollow ugc">GitHub - bdhirsh/pytorch_open_registration_example: Example of using pytorch's open device registration API</a>   I am able to register the device with this repo and added TORCH_LIBRARY_IMPL(_, PrivateUse1, m) { m.fallback(torch::CppFunction::makeFallthrough()); }   but still am not able to run simple torch.add test or torch.negative test, do i need register all the kernels too, ?  i just need annotation to tensors remaining compute should be fine with cpu.   based on tensor annotation i can do mem copies from host to device and device to host</p>
Out of Tree Backend Dispatcher - Registering Structured Ops

neeldug: <p>After following the open registration example for attempting to create an out of tree backend that replicates CUDA but remotely via an API call instead, I’ve managed to perform registration for the AbsKernel using the <code>REGISTER_DISPATCH</code> with the associated DispatchStub, and then used</p>
<pre><code class="lang-auto">at::Tensor &amp;abs_out(const at::Tensor &amp;self, at::Tensor &amp;out) {
  return at::native::abs_out(self, out);
}
</code></pre>
<p>in order to register that operation, but for the Activation function kernels, I cannot seem to replicate the same, for instance trying to use:</p>
<pre><code class="lang-auto">at::Tensor &amp;elu_out(const at::Tensor &amp;self, const at::Scalar &amp;alpha,
                    const at::Scalar &amp;scale, const at::Scalar &amp;input_scale,
                    at::Tensor &amp;out) {
  return at::native::elu_out(self, alpha, scale, input_scale, out);
}
</code></pre>
<p>fails since <code>at::native::elu_out</code> doesn’t exist, any help would be much appreciated!</p>

Tiru_B: <p><a class="mention" href="/u/neeldug">@neeldug</a> , How are you able to run out of tree backend, I am able to register my backend by following this example</p>
<aside class="onebox githubfolder" data-onebox-src="https://github.com/bdhirsh/pytorch_open_registration_example/tree/master">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/bdhirsh/pytorch_open_registration_example/tree/master" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/bdhirsh/pytorch_open_registration_example/tree/master" target="_blank" rel="noopener nofollow ugc">GitHub - bdhirsh/pytorch_open_registration_example: Example of using pytorch's...</a></h3>

  <p><a href="https://github.com/bdhirsh/pytorch_open_registration_example/tree/master" target="_blank" rel="noopener nofollow ugc">master</a></p>

  <p><span class="label1">Example of using pytorch's open device registration API - bdhirsh/pytorch_open_registration_example</span></p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>but now when i am running any ops like torch native ops i am not able to run i have added</p>
<p>TORCH_LIBRARY_IMPL(_, PrivateUse1, m) { m.fallback(torch::CppFunction::makeFallthrough()); }  to my cpp code to fall back cpu device but still am not able run simple add example it self.   do i need to register all the ops for my backend ? how to do this.  All i need is my_backend name as anotater to torch tensors to avoid host to device copies.</p>
Looking for Pytorch Developer

Aglowid_IT_Solutions: <p><strong>Job Responsibilities:</strong></p>
<p>• Develop and maintain robust, scalable, and efficient Python applications.</p>
<p>• Write clean, well-documented, and testable code.</p>
<p>• Collaborate with cross-functional teams to design and implement solutions.</p>
<p><strong>Required Skills:</strong></p>
<p>• Strong proficiency in Python programming language.</p>
<p>• Experience with relevant Python libraries such as NumPy, Pandas, PyTorch/TensorFlow (depending on the specific role/team).</p>
<p>• Expertise in package and version management using Conda, Pip, and Virtualenv.</p>
<p>• Good understanding of Git version control.</p>
<p>• Experience with Docker for containerization.</p>
<p>• Familiarity with Bash scripting.</p>

Jay_G: <p>Hello,</p>
<p>Are you looking for freelance developer? if yes, Please ping me at jay@liquifysolutions.in</p>

Jay_G: <p>My email id is: jaygupta@liquifysolutions.in</p>
Nan values caused by scatter

gitKincses: <p>I am trying to perform some indexing operations on the features of attention calculation. Below is a simple test code that maps the initial feature feat to [feat, feat], then performs<br>
normal attention calculation, and finally inverses it back to the original length. In this process, multiple points are mapped to the same position, so I take the average method.<br>
However, this causes NaN during backpropagation. I checked the feature values in forward propagation each time and found no NaN. How should I correctly perform the above indexing<br>
operations?</p>
<pre><code class="lang-auto">    def forward(self, point): 

        ...

        if int(offset2bincount(point.offset).min()) &lt;= self.patch_sizes[self.stage_idx]:     
            ...
        else:       

            a = torch.arange(0, len(L0)).to(L0.device)
            b = torch.arange(0, len(L0)).to(L0.device)
            L0_mapped = torch.cat([a, b]).to(L0.device)
            L0_sorted = L0[L0_mapped]                   # [feat, feat]. here L0 == feat

            cu_seqlens = torch.arange(1, first_cu_seqlens.shape[0] * 2) * self.patch_sizes[self.stage_idx] 
            cu_seqlens = torch.cat([torch.tensor([0]), cu_seqlens]).to(
                dtype=torch.int32).cuda() 

        H = self.num_heads
        K = self.patch_sizes[self.stage_idx]
        C = self.channels


        q = self.q(L0_sorted)
        kv = self.kv(L0_sorted)


        feat = flash_attn.flash_attn_varlen_kvpacked_func(
                q.half().reshape(-1, H, C // H),
                kv.half().reshape(-1, 2, H, C // H),
                cu_seqlens,
                cu_seqlens,
                max_seqlen_q=self.patch_sizes[self.stage_idx],
                max_seqlen_k=K,
                dropout_p=self.attn_drop if self.training else 0,
                softmax_scale=self.scale,
        ).reshape(-1, C)
        feat = feat.to(q.dtype)

        if int(offset2bincount(point.offset).min()) &gt; self.patch_sizes[self.stage_idx]:

            target_size = len(L0)

            feat_inversed = torch.zeros(target_size, feat.size(1), dtype=feat.dtype, device=feat.device)
            L0_mapped_expanded = L0_mapped.unsqueeze(1).expand(-1, self.channels)

            feat_inversed.scatter_add_(0, L0_mapped_expanded, feat)        # [feat, feat] -&gt; feat
 
            counts = torch.zeros(target_size, feat.size(1), dtype=feat.dtype, device=feat.device)
            counts.scatter_add_(0, L0_mapped_expanded, torch.ones_like(feat))

            feat = feat_inversed / counts          # Avg

        feat = feat[first_inverse]

        # ffn
        ...

        if torch.isnan(feat).any() or torch.isinf(feat).any():
            raise ValueError("feat contains NaN or inf.")


        return point
</code></pre>
How to reset Model parameters to random values after training it?

ahirema3: <p>I have a requirement where I want to reset my model parameters to random values after a constraint is satisfied.</p>
<pre><code class="lang-auto">class SamplerModel(nn.Module):
    def __init__(self, input_size, hidden_size, n_dims):
        super(SamplerModel, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size, hidden_size)
        self.fc_mu = nn.Linear(hidden_size, 1)  # Output layer for mu
        self.fc_sigma = nn.Linear(hidden_size, n_dims)  # Output layer for sigma

    def forward(self, x):
        x = self.relu(self.fc1(x))
        x = self.relu(self.fc2(x))
        mu = self.fc_mu(x)
        sigma = torch.exp(self.fc_sigma(x))  # Apply exponential to ensure sigma is positive
        return mu, sigma

</code></pre>
<p>Above is my model. I want to reset the values after certain constraint satisfaction. how do I do it.</p>

ptrblck: <p>You can call the <code>.reset_parameters()</code> method of each module containing trainable parameters or you could write your custom initialization method and call it via <code>model.apply</code>.</p>

Geremia: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="204818">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>.reset_parameters()</p>
</blockquote>
</aside>
<p>Is there a way to do this with <code>nn.Sequential</code> models?</p>

ptrblck: <p>Yes, you can iterate the submodules:</p>
<pre data-code-wrap="python"><code class="lang-python">model = nn.Sequential(
    nn.Linear(1, 1),
    nn.ReLU(),
    nn.Linear(1, 1)
)

print(model[0].weight)
# Parameter containing:
# tensor([[-0.8953]], requires_grad=True)

for m in model.modules():
    if hasattr(m, "reset_parameters"):
        m.reset_parameters()

print(model[0].weight)
# Parameter containing:
# tensor([[0.9524]], requires_grad=True)
</code></pre>
Redefining model with fewer parameters → out of memory (OOM)?

Geremia: <p>Why when I redefine my model to have fewer parameters and retrain (after training once before) does PyTorch sometimes through an OOM error?</p>
<p>In other words: How do I tune model hyperparameters without running into OOM errors?</p>
<p>Do I have to reset the computational graph / saved gradients in the dataset’s tensors? If so, how?</p>

Geremia: <p>When I make too big of a model and training throws an OOM error when passing an batch to it, I have to <code>del model</code> and run</p>
<pre data-code-wrap="python"><code class="lang-python">gc.collect()
torch.cuda.empty_cache()
</code></pre>
Access to TensorImpl

wonkyoc: <p>I am trying to access a low-level tensor and retrieve its member via SystemTap, such as dim() or sizes(). I know the layout of TensorImpl defined in <code>c10/core/TensorImpl.h</code> but for some reasons, my gdb or stap cannot retrieve its information despite having debug symbols.</p>
<p>I have built PyTorch from the source codes w/ <code>DEBUG=1</code> and I did verify its symbol via <code>nm -D libtorch_cpu.so</code>: _ZTIN3c1010TensorImplE, which is a mangled symbol of c10::TensorImpl.</p>
<p>However, when I track this symbol to print its layout, I hit the following error:</p>
<pre><code class="lang-auto">(gdb) ptype c10::TensorImpl
No type "TensorImpl" within class or namespace "c10".
</code></pre>
<p>I can see other functions as the same way:</p>
<pre><code class="lang-auto">(gdb) ptype c10::intrusive_ptr_target
type = class c10::intrusive_ptr_target {
  private:
    struct std::atomic&lt;unsigned int&gt; refcount_;
    struct std::atomic&lt;unsigned int&gt; weakcount_;

  protected:
    ~intrusive_ptr_target();
    intrusive_ptr_target();
    intrusive_ptr_target(class c10::intrusive_ptr_target &amp;&amp;);
    intrusive_ptr_target(const class c10::intrusive_ptr_target &amp;);
    class c10::intrusive_ptr_target &amp; operator=(class c10::intrusive_ptr_target &amp;&amp;);
    class c10::intrusive_ptr_target &amp; operator=(const class c10::intrusive_ptr_target &amp;);
  private:
    virtual void release_resources();
}
</code></pre>
<p>So, I am quite confused why this is not accessible despite its symbol. If there is no symbol then I get it becasue it may be stripped way from optimization flags; yet, having symbols and not showing its internal layout does not make sense to me.</p>
AttributeError:'tuple' object has no attribute 'size'

Bay: <p>I tried to print out the summary of a simple model I created with LSTM and CNN but I got the error :AttributeError:‘tuple’ object has no attribute ‘size’.<br>
Here is my code :<br>
class lstmCnn_model(nn.Module):</p>
<pre><code>def __init__(self):
    super(lstmCnn_model, self).__init__()
    self.lstm = nn.LSTM(input_size=384, hidden_size=64, batch_first=True, bidirectional=True)
  
    self.layernorm = nn.LayerNorm(128)
    self.conv = nn.Conv1d(in_channels=128, out_channels=64, kernel_size=3)
    
    
def forward(self,x):
    
   
    x = x.view(x.size(0), x.size(2),-1)
    
    x, _ = self.lstm(x)
    
    x = self.layernorm(x)
    
    x = self.conv(x)
    print(x.shape)
    
    
    return x
</code></pre>
<p>model = lstmCnn_model()<br>
model.to(device)<br>
print(model)<br>
summary(model, input_size= (3,128,128))</p>

ptrblck: <p>Your code works for me after fixing the wrong input shape:</p>
<pre data-code-wrap="python"><code class="lang-python">class lstmCnn_model(nn.Module):
    def __init__(self):
        super(lstmCnn_model, self).__init__()
        self.lstm = nn.LSTM(input_size=384, hidden_size=64, batch_first=True, bidirectional=True)
        self.layernorm = nn.LayerNorm(128)
        self.conv = nn.Conv1d(in_channels=128, out_channels=64, kernel_size=3)
        
    def forward(self,x):
        x = x.view(x.size(0), x.size(2),-1)
        x, _ = self.lstm(x)
        x = self.layernorm(x)
        x = self.conv(x)
        print(x.shape)
        return x


model = lstmCnn_model()
print(model)

from torchinfo import summary
summary(model, input_size= (3,128,128))
# RuntimeError: Failed to run torchinfo. See above stack traces for more details. Executed layers up to: []

summary(model, input_size= (3,384,128))
# ==========================================================================================
# Layer (type:depth-idx)                   Output Shape              Param #
# ==========================================================================================
# lstmCnn_model                            [3, 64, 126]              --
# ├─LSTM: 1-1                              [3, 128, 128]             230,400
# ├─LayerNorm: 1-2                         [3, 128, 128]             256
# ├─Conv1d: 1-3                            [3, 64, 126]              24,640
# ==========================================================================================
# Total params: 255,296
# Trainable params: 255,296
# Non-trainable params: 0
# Total mult-adds (M): 97.79
# ==========================================================================================
# Input size (MB): 0.59
# Forward/backward pass size (MB): 0.98
# Params size (MB): 1.02
# Estimated Total Size (MB): 2.59
# ==========================================================================================
</code></pre>

Bay: <p>Thank you. Using summary from torchinfo it work.</p>
Does FSDP shard frozen layers?

tcm: <p>Hi,</p>
<p>I’m studying how to use FSDP effectively from the following video:</p>
<div class="youtube-onebox lazy-video-container" data-video-id="HQeKwCsnH4k" data-video-title="Part 1: Accelerate your training speed with the FSDP Transformer wrapper" data-video-start-time="" data-provider-name="youtube">
  <a href="https://www.youtube.com/watch?v=HQeKwCsnH4k" target="_blank" class="video-thumbnail" rel="noopener nofollow ugc">
    <img class="youtube-thumbnail" src="https://img.youtube.com/vi/HQeKwCsnH4k/maxresdefault.jpg" title="Part 1: Accelerate your training speed with the FSDP Transformer wrapper" width="690" height="388">
  </a>
</div>

<p>In summary, the video teaches how to choose layers to be sharded with <code>transformer_auto_wrap_policy</code>. He mentions that we should shard layer classes that wrap multihead attention and feedforward layers.</p>
<p>I have one question that hasn’t been made clear in the video: When we fine-tune an available model with most of the parameters frozen, can we shard the frozen layers across ranks even if they are not trainable and don’t have gradients and optimizer states?<br>
An example could be an LLM where all layers are frozen and only the LM head is trainable, if we still do what told in the video above, are there still benefits?</p>
<p>Thank you.</p>