akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Oferta Laboral Payton con ML, Pytorch, Torchaudio, Torchvideo

username: NataliaZagalabs
cooked: <p>Una empresa colombiana en crecimiento global busca un/a <strong>Desarrollador/a con experiencia en Machine Learning</strong>, altamente competente en <strong>Python</strong>, para sumarse a un equipo internacional.<br>
Si tienes conocimientos sólidos en <strong>PyTorch, Torchaudio y Torchvideo</strong>, te apasiona la inteligencia artificial aplicada a audio y video, y hablas inglés con fluidez, ¡esta oportunidad es para ti!<br>
<img src="https://discuss.pytorch.org/images/emoji/apple/round_pushpin.png?v=14" title=":round_pushpin:" class="emoji" alt=":round_pushpin:" loading="lazy" width="20" height="20"> Modalidad: Trabajo <strong>100% remoto</strong></p>
<ul>
<li>Contratación desde <strong>Colombia o Argentina</strong></li>
<li>Pagos en <strong>USD</strong></li>
</ul>
<p>CV <a href="mailto:natalia.ramirez@zagalabs.com">natalia.ramirez@zagalabs.com</a></p>

username: ptrblck
cooked: <p>From Google Translate:</p>
<pre><code class="lang-auto">A Colombian company with global growth is seeking a Machine Learning Developer with extensive Python experience to join an international team.
If you have solid knowledge of PyTorch, Torchaudio, and Torchvideo, are passionate about artificial intelligence applied to audio and video, and are fluent in English, this opportunity is for you!
:round_pushpin: Working Mode: 100% remote

Hiring from Colombia or Argentina
Payments in USD```</code></pre>
Error when moving to GPU

username: thistlillo
cooked: <p>Why do I get this error? What should I check? I have been using Torch for years, but I have never met this error:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; torch.cuda.device_count()
1
&gt;&gt;&gt; device = torch.device("cuda:0")
&gt;&gt;&gt; print(device)
cuda:0
&gt;&gt;&gt; t = torch.Tensor([1])
&gt;&gt;&gt; t
tensor([1.])
&gt;&gt;&gt; t = t.to(device)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
RuntimeError: CUDA error: operation not supported
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

&gt;&gt;&gt; torch.__version__
'2.5.0+cu124'
&gt;&gt;&gt; 
</code></pre>

username: ptrblck
cooked: <p>Did you change anything on your system recently, e.g. updated the NVIDIA driver without rebooting?</p>

username: thistlillo
cooked: <p>Thank you, ptrblck: that was the issue.</p>
<p>The drivers were updated after a system failure (I was not aware of) and I created the python environment using the “old” requirement list.</p>
<p>I will never understand how the various systems manage Python packages: I can report zero success cases in using Python environments across machines/OSs <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Fine-tuning open-clip with lora, Loss.backward(), RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

username: vietvo
cooked: <p>I got the same error <code>RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</code> with my code. My notebook here <a href="https://www.kaggle.com/code/vietvo89/finetune-open-clip-torch-peft-randlora" class="inline-onebox" rel="noopener nofollow ugc">Finetune_open_clip_torch-peft-RandLoRA | Kaggle</a>. I tried to fine-tune open-clip but not sure why I got the error although I enabled requires_grad of some parameters. It’s great if you can give me a hand and tell me what’s wrong with my code. Thanks</p>

username: ptrblck
cooked: <p>You could debug this issue by checking where you are detaching the forward activation in the model execution by printing the <code>.grad_fn</code> at various places. This would allow you to narrow down the operation or module which is detaching the computation graph.</p>

username: vietvo
cooked: <p>Thanks for your quick response. I printed <code>result.grad_fn</code> and found that it might be detached from the computation graph. The author of the <code>RandLoRA</code> code already replied to me and updated the code. I did the same thing to print <code>result.grad_fn</code> and this time it was attached to the computation graph.</p>

username: vietvo
cooked: <p>If you don’t mind, could you do me a favor? After addressing this issue, I ran the training loop, but the loss remained unchanged and the model did not update or learn properly. The <code>Linear</code> layer worked correctly but <code>MultiheadAttention</code> did not work. You can produce my results by running the notebook. I appreciate your help. Thanks</p>

username: ptrblck
cooked: <p>Could you describe why you think the MHA layer does not work?</p>

username: vietvo
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, When running the training loop, after each epoch, the loss and accuracy were almost unchanged. I inspected the weights of Lora, which were changed somehow, but eventually, it almost did not affect or improve the model’s performance. So I think MHA might not work. It’s weird to me. I’m not sure what’s wrong with the code.</p>

username: Akilsurya_S
cooked: <p>I think you’re trying to call <code>.backward()</code> on a tensor that <strong>does not track gradients</strong>, i.e., it has <code>requires_grad=False</code> and does not originate from any differentiable computation.</p>
<p>Make sure:</p>
<ul>
<li>LoRA-injected layers have <code>requires_grad=True</code></li>
<li>Model outputs go into a loss function (like <code>nn.CrossEntropyLoss</code>)</li>
<li>The tensor you call <code>.backward()</code> on is part of the computation graph</li>
</ul>
Install PyTorch for all users on Ubuntu

username: zzzhhhzzzhhh
cooked: <p>I want to install PyTorch that can be shared by all users of a Ubuntu workstation. If I followed the command provided by “<a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">Start Locally</a>”:</p>
<blockquote>
<p>pip3 install torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu118" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu118</a></p>
</blockquote>
<p>, the PyTorch package will be installed to <code>.local/lib/python3.10/site-packages/</code>in my own home directory, which cannot be seen by other users. Is there any way to force the installation directory of PyTorch to be in the Ubuntu system (e.g., <code>/usr/lib/python3/dist-packages'</code> which is in <code>sys.path</code>) so that other users of the Ubuntu system could use PyTorch I installed?</p>
<p>My system:</p>
<ul>
<li>Python version: 3.10.12</li>
<li>pip version: 22.0.2</li>
<li>Ubuntu version 22.04.5 LTS</li>
</ul>

username: ptrblck
cooked: <p>The install location of Python packages depends on how your Python environment was configured and where the Python interpreter is located. E.g. you could install Python in a “global” folder (e.g. <code>/opt</code>) which would then be accessible for all users.</p>

username: zzzhhhzzzhhh
cooked: <p>Is it possible that pytorch has some special code that directs pip3 to forcibly install pytorch package to <code>site.USER_SITE</code> (as if I use <code>--user</code> option but I didn’t)?</p>

username: zzzhhhzzzhhh
cooked: <p>The reason is that I forgot to add <code>sudo -H</code> before the given command, so pip falls back to <code>site.USER_SITE</code> because I don’t have permission to write system directory. Now I add “sudo -H” and pytorch is installed to <code>/usr/local/lib/python3.10/dist-packages</code> as desired.</p>
<p>Thanks for reading and replying my question.</p>
[Notice] Ascend Community Meeting Announcement

username: jayggh
cooked: <p><span class="hashtag-raw">#privateuse1</span><br>
Community meetings will be organized every Wednesday at 2:30 PM starting from June 11; the meeting content includes:</p>
<ol>
<li>Torch_NPU roadmap and progress explanation</li>
<li>Torch_NPU requirement collection</li>
<li>Progress and discussion on PyTorch native community contributions</li>
<li>Progress on PyTorch third-party ecosystem support</li>
<li>Collection of PyTorch third-party ecosystem requirements and cooperation suggestions</li>
<li>Meeting link will be released later, please stay tuned</li>
</ol>

username: jayggh
cooked: <p>You can join via this link [<a href="https://app.huawei.com/wmeeting/join/96782654/lLLPyH6JCxQo8XgFQqq2a5avhf2HZBJHZ#success" class="inline-onebox" rel="noopener nofollow ugc">Join meeting</a>]</p>

username: jayggh
cooked: <p>The time is changed to 4:00 PM Beijing time</p>
YOLO + ViTPose + LSTM for sequence classification

username: Soham_Bhaumik
cooked: <p>Hello. I am trying to create a simple skeleton based posture recognition system for animals. i have fine tuned a yolo model. for every frame of a video, i detect a dog using yolo and then compute 2D skeletal keypoints using ViTpose. Then i compute the following features:</p>
<ol>
<li>bounding box aspect ratio</li>
<li>all keypoint coordinates normalized with respect to the bounding box center.</li>
<li>some joint angles (unsigned) (using torch.cosinesimilarity)</li>
<li>some keypoint distances(signed, normalized wrt to bounding box).<br>
to factor in occlusions and low conf predictions, i create a mask array of 1’s and 0’s. this is of the same shape as the features.<br>
in my custom dataset class, i have:</li>
</ol>
<pre><code class="lang-auto">def create_overlapping_sequences(self, features, masks, class_idx):
        num_frames, num_features = features.shape[0], features.shape[1]
        sequences = []
        labels = []
        mask_sequences = []
        for frame_index in range(0, num_frames - (self.sequence_length - self.stride), self.stride):
            sequences.append(features[frame_index : frame_index + self.sequence_length])
            mask_sequences.append(masks[frame_index : frame_index + self.sequence_length])
            labels.append(class_idx)
        return sequences, mask_sequences, labels
</code></pre>
<p>to create overlapping sequences to sort of augment data. and then in the lstm classifier, i do:</p>
<pre><code class="lang-auto">def forward(self, x, mask=None):
        """
        Forward pass
        
        Args:
            x: Input tensor of shape (batch_size, sequence_length, num_features)
        
        Returns:
            logits: Output tensor of shape (batch_size, num_classes)
        """
        batch_size, seq_len, num_features = x.shape
        
        if mask is not None:
            x = x * mask
        
        x = self.input_projection(x)
        
        lstm_out, (hidden, cell) = self.lstm(x)
        last_output = lstm_out[:, -1, :]
        logits = self.classifier(last_output)
        
        return logits
</code></pre>
<p>i am trying to handle occluded features while masking. for example, while calculating angle between 3 joints, if i get all kpt’s visibility as 1, i add the value in degrees to my features array and add a mask 1 in mask array. if even one kpt has visibility 0, i add mask 0 and feature -999 (a large value). same for all other features including normalized kpt coordinates (normalized with bbox center) and inter-joint distances.</p>
<p>no matter what i do, this classifier keeps putting out the same label for all postures. how do i go about solving this problem? have been at it for 2 days now. at my wits end</p>
Weighted Convolution class

username: cammarasana123
cooked: <p>Dear all,<br>
I would like to share my contribution with the class: weighted convolution.<br>
(<a href="https://github.com/cammarasana123/weightedConvolution2.0" class="inline-onebox" rel="noopener nofollow ugc">GitHub - cammarasana123/weightedConvolution2.0: Weighted Convolution 2.0</a>)</p>
<p>This class applies a density function to scale the contribution of neighbouring pixels based on their distance from the central pixel, with a computational cost comparable to standard convolution.</p>
<p>I would be glad to get a feedback from the community,</p>
<p>Simone</p>
How to interpret the output of CommDebugMode

username: githubsgi
cooked: <p>How should the following be interpreted ?</p>
<blockquote>
<p>FORWARD PASS<br>
*c10d_functional.all_reduce: 2<br>
**aten._foreach_norm.Scalar<br>
**aten._foreach_norm.Scalar<br>
**aten.stack.default<br>
**aten.div.Tensor<br>
**aten.div.Tensor<br>
**aten.div.Tensor<br>
**aten.div.Tensor<br>
**aten.stack.default<br>
**aten.linalg_vector_norm.default<br>
shape: [torch.Size([13])]<br>
sharding: [(_NormPartial(reduce_op=‘sum’, norm_type=1.0), _NormPartial(reduce_op=‘sum’, norm_type=1.0))]<br>
device mesh: DeviceMesh(‘cuda’, [[0, 1], [2, 3]], mesh_dim_names=(‘dp’, ‘tp’))<br>
**aten.linalg_vector_norm.default<br>
**aten.add.Tensor<br>
shape: [torch.Size(<span class="chcklst-box fa fa-square-o fa-fw"></span>)]<br>
sharding: [(_NormPartial(reduce_op=‘sum’, norm_type=1.0), _NormPartial(reduce_op=‘sum’, norm_type=1.0))]<br>
device mesh: DeviceMesh(‘cuda’, [[0, 1], [2, 3]], mesh_dim_names=(‘dp’, ‘tp’))<br>
**aten.add.Tensor<br>
**aten.reciprocal.default<br>
shape: [torch.Size(<span class="chcklst-box fa fa-square-o fa-fw"></span>)]<br>
sharding: [(_NormPartial(reduce_op=‘sum’, norm_type=1.0), _NormPartial(reduce_op=‘sum’, norm_type=1.0))]<br>
device mesh: DeviceMesh(‘cuda’, [[0, 1], [2, 3]], mesh_dim_names=(‘dp’, ‘tp’))<br>
**_c10d_functional.all_reduce.default<br>
**_c10d_functional.all_reduce.default<br>
**_c10d_functional.wait_tensor.default<br>
**aten.reciprocal.default<br>
**aten.mul.Tensor<br>
shape: [torch.Size(<span class="chcklst-box fa fa-square-o fa-fw"></span>)]<br>
sharding: [(Replicate(), Replicate())]<br>
device mesh: DeviceMesh(‘cuda’, [[0, 1], [2, 3]], mesh_dim_names=(‘dp’, ‘tp’))<br>
**aten.mul.Tensor<br>
**aten.clamp.default<br>
shape: [torch.Size(<span class="chcklst-box fa fa-square-o fa-fw"></span>)]<br>
sharding: [(Replicate(), Replicate())]<br>
device mesh: DeviceMesh(‘cuda’, [[0, 1], [2, 3]], mesh_dim_names=(‘dp’, ‘tp’))<br>
**aten.clamp.default<br>
**aten.<em>foreach_mul</em>.Tensor<br>
shape: [torch.Size(<span class="chcklst-box fa fa-square-o fa-fw"></span>)]<br>
sharding: [(Replicate(), Replicate())]<br>
device mesh: DeviceMesh(‘cuda’, [[0, 1], [2, 3]], mesh_dim_names=(‘dp’, ‘tp’))<br>
**aten.<em>foreach_mul</em>.Tensor</p>
</blockquote>
Bicubic or tricubic interpolation on 3d tensors

username: hardik_rathod
cooked: <p>I am wanting to standardize 3d tensors to shape (32,512,512) which is the (depth, height, width) using <code>bicubic</code> or <code>tricubic</code> mode for interpolation.</p>
<p>I am aware that bicubic mode is present in torch.nn.functional.interpolate as per the <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.interpolate.html" rel="noopener nofollow ugc">documentation</a><br>
<code><br>
The modes available for resizing are: nearest, linear (3D-only), bilinear, bicubic (4D-only), trilinear (5D-only), area, nearest-exact<br>
</code></p>
<p>But its only for 4D data. Is there a way to implement with bicubic or tricubic interpolation on 3d tensors</p>

username: ptrblck
cooked: <blockquote>
<p>Currently temporal, spatial and volumetric sampling are supported, i.e. expected inputs are 3-D, 4-D or 5-D in shape.</p>
</blockquote>
<p>So I assume volumetric refers to a shape of <code>[batch_size, channels, depth, height, width]</code>.</p>

username: hardik_rathod
cooked: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
thanks for your reply but, I am unable to understand how it helps my problem.</p>
<p>I just want to understand if there is a way in which I can implement <code>Bicubic</code> transformation on 3d data points.<br>
The current blocker I am facing is that the available bicubic transformation is present for 4d data points.</p>
<p>Another option is if I could add another dimension to my 3d data to make it 4d and then implement the available bicubic interpolation. But, the problem with that is if i make my data 4d then the interpolate function will add 2 more dimensions to it for batch and Num_channels which will make it 6d, which is not acceptable</p>
<p>Please let me know your thoughts on this</p>
<p>Thanks</p>

username: Tarek_Alkabbani
cooked: <p>Hey <a class="mention" href="/u/hardik_rathod">@hardik_rathod</a>,</p>
<p>I am having the same issue as you. Bicubic interpolation is only supported for 2D images but not 3D images. Did you end up figuring it out or did you have to write your own implementation?</p>

username: Hesam_Tofighian
cooked: <p>Hi <a class="mention" href="/u/tarek_alkabbani">@Tarek_Alkabbani</a>,<br>
I am also intersted. Have you impelemented cubic or any higher order interpolation in pytorch for 3D spatial data (5D in shape)?</p>
What options do I have when torch.compile with TF32 matmuls causes accuracy issues?

username: awang_1x
cooked: <p>Hi there, I have a model where torch.compile() with <code>torch.set_float32_matmul_precision("highest")</code> has no accuracy issues, but with <code>torch.set_float32_matmul_precision("high")</code> or lower has accuracy issues. <a href="https://blog.ezyang.com/2024/11/ways-to-use-torch-compile/" rel="noopener nofollow ugc">As I understand</a>, there is no guarantee that the numerics will not differ from eager, so I’m looking for a workaround.  Does torch.compile have any built in way or decorator where I can disable/turn off TF32 for some of the modules or functions in the <code>nn.Module</code> so I can close the accuracy gap?</p>
Why does GPU memory usage not double when loading two identical models in PyTorch?

username: ximen0806449
cooked: <p>Hi everyone,</p>
<p>I have a question regarding GPU memory usage when loading identical models in PyTorch.</p>
<p>I am working on a project where I need to load the same model twice onto the GPU. However, I noticed that when I load the second model, the GPU memory usage only increases slightly, instead of doubling as I expected.</p>
<p>Why does PyTorch share memory between identical models? Is this an optimization feature or a limitation of how PyTorch handles GPU memory?</p>
<p>I would greatly appreciate it if someone could explain this behavior in detail or point me to relevant resources.</p>

username: ximen0806449
cooked: <p>Thanks for your reply!</p>
<p>I attempted to use threading to load two identical models, where one handles <code>model_1</code> and the other handles <code>model_2</code>. Interestingly, I noticed that the GPU memory usage doubled. Could you please explain what might be causing this phenomenon?</p>

username: ptrblck
cooked: <p>The previous response was a wrong claim about parameter sharing created by a bot.<br>
How do you measure the memory usage? Are you checking the allocated memory via <code>torch.cuda.memory_allocated()</code> or including the cached memory?</p>
Num_workers does not run in parallel

username: whoab
cooked: <pre><code class="lang-auto">from torch.utils.data import Dataset, DataLoader
import time
import multiprocessing as mp
import torch

class Sleep(Dataset):
    def __len__(self): return 20
    def __getitem__(self, i):
        import time, os
        time.sleep(1)
        return os.getpid()

if __name__ == "__main__":
    mp.set_start_method("fork", force=True)

    loader = DataLoader(Sleep(), batch_size=20, num_workers=10, persistent_workers=True)

    t0 = time.time()
    next(iter(loader))
    print("wall = ", time.time() - t0)   # should be ≈ 3-4 s, not 180 s

    next(iter(loader))
    print("wall = ", time.time() - t0)   # should be ≈ 3-4 s, not 180 s

    next(iter(loader))
    print("wall = ", time.time() - t0)   # should be ≈ 3-4 s, not 180 s

    import pdb; pdb.set_trace()
</code></pre>
<p>I have this simple test script. On different systems, with at least 10 cpus each, I run it - each time, each batch takes 20 seconds to load, meaning things are running <em>serially</em> and not in parallel with multiprocessing. Why? How do we actually parallelize the dataloader?</p>

username: ptrblck
cooked: <p>That’s expected since each worker will load the entire batch (i.e. 20 samples a 1 sec loading time each). You are recreating the iterator for each test, which will repeat this process.</p>
Using torch rpc to connect to remote machine

username: mndl1
cooked: <p>Hi, I have a use case where I want to train a model for which some layers are on a remote client machine, and most of the layers are on a GPU server. I understand that the <code>torch.distributed.rpc</code> library is there to help with this, but for all of the tutorials I see, there is no example of connecting to a remote host/port, and all the examples are intra-machine. Has anyone tried doing something similar before, or is there some tutorial/code snippets someone can point me towards?</p>

username: patataman
cooked: <p>You have to use <a href="https://pytorch.org/docs/stable/elastic/run.html" rel="noopener nofollow ugc">torchrun</a> to connect them. I had the same problem and also asked about it (<a href="https://discuss.pytorch.org/t/rpc-torchrun-hangs-in-processgroupgloo/197093" class="inline-onebox">RPC + Torchrun hangs in ProcessGroupGloo</a>). In my question you have a minimal example and also the torchrun command</p>
<p>EDIT: Omg, I’m seeing now your question was from 2022 <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=12" title=":joy:" class="emoji" alt=":joy:" loading="lazy" width="20" height="20"></p>

username: mndl1
cooked: <p>No worries, and thanks for the reply! Better late than never!</p>
Validation spike during partial fine-tuning

username: anaf_sousa
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/c/ac9877df8ced62acb80c88694de6e77a964fae6c.png" data-download-href="https://discuss.pytorch.org/uploads/default/ac9877df8ced62acb80c88694de6e77a964fae6c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ac9877df8ced62acb80c88694de6e77a964fae6c_2_690x295.png" alt="image" data-base62-sha1="oCQMLRWltGpQiUCfS8e9HObceEQ" width="690" height="295" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ac9877df8ced62acb80c88694de6e77a964fae6c_2_690x295.png, https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ac9877df8ced62acb80c88694de6e77a964fae6c_2_1035x442.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/c/ac9877df8ced62acb80c88694de6e77a964fae6c_2_1380x590.png 2x" data-dominant-color="F9FAFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1840×789 67 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Hello everyone.<br>
I am training a ResNet50 model (pre-trained on RadImageNet’s Pytorch’s weights) following a partial fine-tuning approach (only the last residual block and FC layer are trainable).<br>
I am working with 525 training samples, using learning rate = 1e-4; batch size = 8; and Adam optimizer.<br>
As you can see in the plot, in the first few epochs, the training accuracy increases to approximately 100%, while the loss efficiently decreases and is close to zero. However, during validation, there is a spike in the initial epochs.<br>
Have you ever observed a similar behaviour with Transfer Learning? Can it be due to batch normalization layers? What can I try to avoid such behaviour? Is PyTorch’s ReduceLROnPlateau an option to consider? I am afraid that, if I use this technique, the LR becomes too small for the model to converge.<br>
Thank you.</p>
Does pytorch descriptor hide real gpu memory address of tensor?

username: namch013633
cooked: <p>Hello, I’m recently working on project and it requires for me to know the gpu memory address of input tensor and output tensor for convolution operation.</p>
<p>First, I checked the weight address of model like below</p>
<p>for name, param in model.named_parameters():<br>
model_param_dict[name] = param.data_ptr()</p>
<p>Then, I captured a cudagraph like below and then extracted kernelParams like below</p>
<p>stream = torch.cuda.Stream()<br>
with torch.cuda.stream(stream):<br>
err = cudart.cudaStreamBeginCapture(stream.cuda_stream, cudart.cudaStreamCaptureMode.cudaStreamCaptureModeGlobal)<br>
model(inputs)<br>
err, graph = cudart.cudaStreamEndCapture(stream.cuda_stream)</p>
<p>with torch.cuda.stream(stream):<br>
err, _, num_nodes = cudart.cudaGraphGetNodes(graph, num_nodes)<br>
err, nodes, num_nodes = cudart.cudaGraphGetNodes(graph, num_nodes)<br>
kernel_params = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for node in nodes:<br>
param_addresses = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
if node.kernelParams:<br>
param_array = ctypes.cast(node.kernelParams, ctypes.POINTER(ctypes.c_void_p))</p>
<pre><code>        # NULL이 나올 때까지 모든 파라미터 주소 출력
        param_idx = 0
        while True:
            addr = param_array[param_idx]
            if addr:
                param_addresses.append(f"{hex(addr)}") 
                param_idx += 1
            else:
                break
        
        kernel_params.append(param_addresses)
</code></pre>
<p>as far as  I know, kernel_params list must contain the elements of model_param_dict since kernel parameters must contain weight address. HOWEVER, kernel_params list only contain some subsets of model_param_dict. It seems like kernel_params only have the weight address when the convolution algorith  is winograd. Can someone explain what is the reason of it and how can I find a weight tensor address for all convolution algorithm with pytorch cnn model?</p>
<p>Thank you in advance!</p>
More data than neurons with autograd?

username: ilja
cooked: <p>Is it possible to use autograd or “manual” back propagation when input is larger than number of neurons and/or when input requires processing outside of nn? For example something like this to find average of N numbers that would be output by one neuron:</p>
<pre><code class="lang-auto">import numpy
import torch
samples = 5
data = numpy.random.default_rng().uniform(0, 1, samples)
reference = numpy.mean(data)
pred = torch.randn(1, requires_grad = True)
lr = 0.01
for i in range(10):
	distance = torch.tensor(
		numpy.pow(pred.item() - reference, 2),
		requires_grad = True)
	distance.backward(retain_graph = True)
	pred.sub_(lr * pred.grad.data)
	pred.grad.data.zero_()
</code></pre>
<p>Above code doesn’t work but I’m not sure how to fix it, or if it’s even possible with autograd/pytorch.</p>

username: ilja
cooked: <p>Managed to make this work with mygrad:</p>
<pre><code class="lang-auto">import numpy
import mygrad
samples = 5
data = numpy.random.default_rng().uniform(0, 1, samples)
reference = mygrad.tensor(numpy.mean(data))
pred = mygrad.tensor(numpy.random.default_rng().uniform(0, 1, 1)[0])
lr = 0.1
for i in range(1):
	distance = numpy.pow(pred - reference, 2)
	distance.backward()
	pred -= lr * pred.grad
</code></pre>

username: Soumya_Kundu
cooked: <p>The below code is working for me</p>
<pre data-code-wrap="python"><code class="lang-python">import numpy
import torch
samples = 5
data = numpy.random.default_rng().uniform(0, 1, samples)
reference = numpy.mean(data)
pred = torch.randn(1, requires_grad = True)
lr = 0.01

for i in range(10):
    distance = torch.pow(pred - reference, 2)
    distance.backward()
    pred.data.sub_(lr * pred.grad.data)
    pred.grad.data.zero_()
</code></pre>
<p>Am I missing something?</p>

username: ilja
cooked: <p>Thanks, looks like I was missing <code>data</code> in <code>pred.data.sub_</code>.</p>
One question about inplemention of Node class of AD module

username: 15800436469try
cooked: <p>I am studying the source code of pytorch automatic differentiation. I have a question to ask. For the Node class in variable.h, as the comment says, in order to avoid circular references in leaf nodes, there is not only one grad_fn on the Node, but also a weak_ptr representing AccumulateGrad. Why not use grad_fn for both, and then add a weak reference to the leaf tensor in the implementation of AccumulateGrad? In this way, can it be unintegrated?</p>
What Cloud Plattform do you use to train your Pytorch models?

username: Miguel_Campos
cooked: <p>Hello,</p>
<p>I know this is maybe not the right place to ask this question. However, I think many people (with extensive experience) can be found in this forum, and they can help with some advice about what platform they use to train their Pytorch models in computer vision.</p>
<p><strong>I wonder what platform you currently use to train machine learning models for computer vision.</strong></p>
<p>I have tried Linode and GCP (Google Cloud Plattform) with good results. However, I find it rare that Linode (which provides an RTX Quatro 4000) offers the same speed (or even faster) than GCP (using its A100). This makes me think that other unspecified hardware (such as the brand of RAM or CPU) comes into the equation.</p>
<p>So, I started looking at other platforms, like Paperspace, Lamda, and Amazon, to train my models quickly and fast with the best cost. <strong>I am not interested in deployment services or even databases.</strong></p>
<p>I am open to discussing your recommendations and personal opinion. I think this thread can be very productive and helpful for other people since the goal is to be able to develop our projects with Pytorch.</p>
<p>PS: I have categorized this topic under Vision because GPU RAM specs may differ from NPL models for vision models.</p>

username: yeldarb
cooked: <p>Hey, ran across this thread &amp; wanted to mention Roboflow as an option. We do cloud GPU training of all sorts of common computer vision models from the YOLO and ResNet families to ViTs and DETRs (like RF-DETR), and VLMs (including SmolVLM, Qwen, and others).</p>
<p>There are a few benefits to training on a platform vs in your own notebook or VM (though we support that as well):</p>
<ul>
<li>You can train on well-tested, battle-hardened, production infra - often, when ML models break they break in subtle ways that don’t error out but just make your model perform worse.</li>
<li>Your engineering time is spent on improving your model &amp; application vs on your MLOps pipeline.</li>
<li>You don’t have to pay for idle GPU time while you develop your pipeline.</li>
<li>It lets you easily spin up many tests in parallel.</li>
<li>The artifacts are automatically created, captured, and registered in the cloud for you.</li>
<li>You can benefit from automatic cloud connections for things like training stats and model evaluation.</li>
<li>It automatically takes advantage of spot instances (and handles automatic recovery from termination).</li>
</ul>
<p>We offer a generous free-tier &amp; extremely advantageous pricing relative to rolling your own, given the above.</p>
<p>(Disclaimer: Roboflow co-founder &amp; CTO)</p>
Vision gaussian formula

username: reinforced
cooked: <p>I’m not sure this is the right place</p>
<pre><code class="lang-auto">def compute_gaussian_weight(pixel_coord, point_mean, inverse_covariance):
    diff = point_mean - pixel_coord
    exponent = -0.5 * diff @ inverse_covariance @ diff.T
    return torch.exp(exponent).item()
</code></pre>
<p>gausssian splatting uses gaussian formula. gaussain formula use inverse_covaraince. It means it does not use euclidean distance?(except for diff)</p>
Why is a distribution object made not subscriptable?

username: tor
cooked: <p>I need to take a subset of a distribution object, then<br>
compare the subset with another subset of another distribution object.<br>
Because the distribution object is not subcriptable (so we can not index directly the distribution object),<br>
my approach so far is to index the member variable, eg probs,<br>
then reconstruct the distribution object later.</p>
<p>Is there any better way for such use-case?<br>
And why is the distribution object made not subscriptable?</p>
<pre><code class="lang-auto">m = torch.distributions.Categorical(probs=torch.rand(5,3))
subset = m.probs[0:2] # Okay
subset = m[0:2] # what I mean with directly indexing the object, but: TypeError: 'Categorical' object is not subscriptable
</code></pre>

username: Zhylkaaa
cooked: <p>I want to ping this thread, because I think distributions module gains more popularity in time-series forecasting right now, and I also have faced similar issue recently. I would much rather subscript the distribution before performing <code>.log_prob</code> for loss, instead of plaing around with mocking and then masking the returned values risking to run into domain constraints and so on.<br>
One use case is missing or “ignore” target value that needs to be handled before computing probs and then masked back, which is not nice.</p>
<p>I don’t see any reason why tensor-style indexing can’t be implemented in those objects? I will be willing to implement it if there is an interest to merge this functionality!</p>
Gemma 3 throws RuntimeError CUDA misaligned address

username: msi-sbraun-11
cooked: <p>Hi there,</p>
<p>I am trying to use Gemma 3 12b it model to generate QA pairs. The pipeline is defined as follows:</p>
<pre><code class="lang-auto">model_id = "google/gemma-3-12b-it" # google/gemma-3-12b-it

    bnb_config = BitsAndBytesConfig(
        load_in_4bit=True
    )

    model = AutoModelForCausalLM.from_pretrained(
        model_id,
        torch_dtype = torch.float32,
        device_map="cuda",
        quantization_config=bnb_config
        )
    tokenizer = AutoTokenizer.from_pretrained(model_id)

    if tokenizer.pad_token is None:
        eos_token_id = model.config.eos_token_id
        eos_token = tokenizer.decode(eos_token_id)
        tokenizer.pad_token = eos_token  # this is a string, which is expected

    text_gen_pipeline = pipeline(
        "text-generation",
        model=model,
        tokenizer=tokenizer,
        max_new_tokens=512,
        torch_dtype=torch.float32, 
        top_p = 0.95,
        top_k = 70,
        temperature = 1.25,
        do_sample=True,
        repetition_penalty=1.3,
    )

    llm = HuggingFacePipeline(pipeline=text_gen_pipeline)

    model = ChatHuggingFace(llm=llm)
</code></pre>
<p>When I use this model using <code>invoke</code> function, at some point it threw an error:</p>
<pre><code class="lang-auto">  File "/home/nokia-proj/miniconda3/envs/vrag/lib/python3.10/site-packages/transformers/integrations/sdpa_attent
ion.py", line 54, in sdpa_attention_forward
    attn_output = torch.nn.functional.scaled_dot_product_attention(
RuntimeError: CUDA error: misaligned address
</code></pre>
<p>Any ideas why this error was encountered and how to resolve this?</p>
<p>Thank you!</p>
<p>PS: Could prompt format be an issue? My prompt is currently like this (when I say ‘like this’, I here mean that there is no indentation for the prompt):</p>
<pre><code class="lang-auto">"""

You will be given a context and a question.

Your task is to provide a 'total rating' scoring how well one can answer the given question unambiguously with the given context.

Give your answer on a scale of 1 to 5, where 1 means that the question is not answerable at all given the context, and 5 means that the question is clearly and unambiguously answerable with the context.

Provide your answer as follows:

Answer:::

Evaluation: (your rationale for the rating, as a text)

Total rating: (your rating, as a number between 1 and 5)

You MUST provide values for 'Evaluation:' and 'Total rating:' in your answer.

Now here are the question and context.

Question: {question}\n

Context: {context}\n

Answer::: """
</code></pre>

username: ptrblck
cooked: <p>Could you add the missing parts of your code making it executable?<br>
Also, which PyTorch version are you using and are you able to reproduce this error in the latest nightly or stable release?</p>
SegFormer Input Size Consistency: RandomResizedCrop vs Resize

username: Sebastien_Grand
cooked: <p>Hi all, I’m new to PyTorch and currently training a SegFormer model on the Cityscapes dataset.</p>
<p>While experimenting with a custom augmentation pipeline I found on GitHub, I noticed something curious: using <code>RandomResizedCrop</code> for training data and <code>Resize</code> for validation data leads to inconsistent input/mask shapes.</p>
<ul>
<li>My original images are of shape <strong>(1024, 2048)</strong>.</li>
<li>For training, I’m using <code>RandomResizedCrop(size=(512, 512), scale=(0.5, 2.0))</code>.</li>
<li>For validation, I use <code>Resize(size=(512, 512))</code>.</li>
</ul>
<p>After augmentation:</p>
<ul>
<li>Training images/masks are of shape <strong>(512, 512)</strong>.</li>
<li>Validation images/masks become <strong>(512, 1024)</strong> — presumably due to the aspect ratio being preserved during resizing.</li>
</ul>
<p>I’ve read that maintaining aspect ratio is important for semantic segmentation, but I’m now wondering:<br>
<strong>How important is it to also maintain consistent input sizes between training and validation?</strong> Will this difference in shape negatively affect evaluation or model performance?</p>
<p>Thanks in advance!</p>

username: Eduardo_Lawson
cooked: <p>I couldn`t reproduce this results on colab.<br>
But for semantic segmentation is very important that you keep the aspect ratio, so the model can get the patterns and learn the features of the dataset. The model learns global and local features, so if you get different features it may perform poorly. The best experiment is to try both ways while keep tracking on the metrics, and after that you will have the real answer, the results will depend on what your data looks like too.</p>

username: Sebastien_Grand
cooked: <p>Thanks a lot, I got the idea, the important is more on keeping aspect ratio than keeping same input shape at the end.</p>
Add custom distributed backend

username: mEm
cooked: <p>Hi everyone,<br>
I would like to know how to add a custom Torch distributed backend.<br>
I added the C++ sources next to the already existing ones (ProcessGroupMPI.cpp and others), but they do not seem to be automatically included (I guess in the <code>TORCH_SRCS</code> cmake variable via a <code>file(GLOB ...)</code> mechanism).<br>
Can someone suggest where I am being wrong?</p>
<p>EDIT: basically the issue is that CMake is not able to see that I added some .cpp sources to the tree, but it correctly added the target link dependency.</p>

username: mEm
cooked: <p>Found it, had to add an entry in <code>build_variables.bzl</code>.</p>

username: Pegah_Alipoormolabas
cooked: <p>Hi, I was looking for a guide on how to implement a custom backend for Torch distributed and I found your post here. Would you mind sharing the resources you used to implement the backend? Thanks</p>
Question about using pytorch-tcn vs. keras-tcn

username: User9393
cooked: <p>Hi everyone <img src="https://discuss.pytorch.org/images/emoji/apple/waving_hand.png?v=14" title=":waving_hand:" class="emoji" alt=":waving_hand:" loading="lazy" width="20" height="20"></p>
<p>I’m currently experimenting with Temporal Convolutional Networks (TCN) and I wanted to ask:</p>
<p>Has anyone here used the pytorch-tcn implementation?<br>
Is it considered reliable or effective for sequence modeling tasks?</p>
<p>Also, I tested the same model architecture and hyperparameters using:<br>
•	keras-tcn<br>
•	pytorch-tcn</p>
<p>However, I got noticeably different results between the two.</p>
<p>What could be the cause of this mismatch?</p>
<p>Any insights or tips would be greatly appreciated!</p>
<p>Thanks</p>
StreamWriter writing wrong HLS format

username: phillies
cooked: <p>Hi,<br>
I have a use case where I need to write a stream of images as HLS stream to disk. The resulting m3u8 file seems wrong, as it always only contains the last 5 segments.</p>
<p>Here’s a simple toy example:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torio
from torchaudio.io import StreamWriter

# Parameters
width, height = 800, 450
frame_rate = 10
num_frames = 300  # 30 seconds of video
manifest_file_path = "output.m3u8"
stream_format = "hls"
codec = "libx264"
pixel_format = "rgb24"
encoder_format = None
gop_size = 2
encoder_opts = {
    "g": str(int(gop_size * frame_rate)),
}
codec_config = torio.io.CodecConfig()
codec_config.gop_size = int(gop_size * frame_rate)


print(
    f"Writer: starting stream writer with format {stream_format}, width {width}, height {height}, "
    f"frame rate {frame_rate}, pixel format {pixel_format}, codec {codec}, "
    f"encoder format {encoder_format}, encoder options {encoder_opts}, codec config {codec_config}, "
    f"manifest file {manifest_file_path}"
)
writer = StreamWriter(manifest_file_path, format=stream_format)
writer.add_video_stream(
    width=width,
    height=height,
    frame_rate=frame_rate,
    format=pixel_format,
    encoder=codec,
    encoder_format=encoder_format,
    hw_accel=None,
    encoder_option=encoder_opts,
    codec_config=codec_config,
)

writer.open()

for _ in range(num_frames):
    frame = torch.randint(0, 256, (1, 3, height, width), dtype=torch.uint8)
    writer.write_video_chunk(0, frame, pts=None)

writer.close()
print(f"Video written to {manifest_file_path}")
</code></pre>
<p>This writes 14 <code>.ts</code> files with the video content but the <code>output.m3u8</code> only contains the last 5 segments:</p>
<pre><code class="lang-auto">#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:2
#EXT-X-MEDIA-SEQUENCE:9
#EXTINF:1.900000,
output9.ts
#EXTINF:1.900000,
output10.ts
#EXTINF:1.900000,
output11.ts
#EXTINF:1.900000,
output12.ts
#EXTINF:1.300000,
output13.ts
#EXT-X-ENDLIST
</code></pre>
<p>So the player only sees the last 8 of 30 seconds in this example.</p>
<p>Am I using it wrongly or is this a bug?</p>
Trace/Retrieve C++ implementations from a single python operation

username: nautsimon
cooked: <p>I am trying to find an easy way to see all the C++ code touched from a python operation.</p>
<p>For example when executing torch.add(x, y), I want to output a single file with either a list of the filenames/function names executed or all the actual C++ backend implementation for that operation.</p>
<p>I have some ideas of how to do this, such as with gdb, llvm-cov, building pytorch with breakpoints, etc. Has anyone done this before? and what tools did you use?</p>
NCCL+Torch Distributed Error

username: yuyuanchen848
cooked: <p>Hello Everyone, When I use deepspeed and I got the error below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b5031e5b7d661b02c89c465060a356e6a7919cd0.png" data-download-href="https://discuss.pytorch.org/uploads/default/b5031e5b7d661b02c89c465060a356e6a7919cd0" title="940a394c8301fe6ca8369a7fb6fb9d6"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b5031e5b7d661b02c89c465060a356e6a7919cd0.png" alt="940a394c8301fe6ca8369a7fb6fb9d6" data-base62-sha1="pPj6qzcd9CQZuS3QX4qNccQBMgo" width="690" height="201" data-dominant-color="2C2C2C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">940a394c8301fe6ca8369a7fb6fb9d6</span><span class="informations">839×245 11.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
And My machine is 5090, cuda 12.2 torch2.6.0+cu126<br>
How can I fix this error?</p>

username: H-Huang
cooked: <p>Can you run this script so we get a full view of your environment setup? <a href="https://github.com/pytorch/pytorch/blob/main/torch/utils/collect_env.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/utils/collect_env.py at main · pytorch/pytorch · GitHub</a></p>
<p>This error likely suggests some CUDA and NCCL incompatibility, also if you’re using cuda12.2 but torch2.6.0+cu126, then torch might be using some features that are part of cuda 12.6. You should use the version with cuda 11.8 compat (cu118)</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="yuyuanchen848" data-post="1" data-topic="220496">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yuyuanchen848/48/75092_2.png" class="avatar"> yuyuanchen848:</div>
<blockquote>
<p>And My machine is 5090, cuda 12.2 torch2.6.0+cu126<br>
How can I fix this error?</p>
</blockquote>
</aside>
<p>Blackwell GPUs require CUDA 12.8 so install the latest stable (2.7.0) or nightly binary and it should work.</p>
Potential Bug with HYBRID_SHARD and (n, 1) Device Mesh Falling Back to NO_SHARD

username: yrraadi-bio
cooked: <p>Hi all,</p>
<p>I ran into unexpected behavior when using FSDP with <code>ShardingStrategy.HYBRID_SHARD</code> and a device mesh of shape <code>(n, 1)</code>, where <code>n</code> is the replicate dimension and <code>1</code> is the (non-existent) shard dimension.</p>
<p>As expected, FSDP falls back to using <code>NO_SHARD</code> in this case, since sharding over a size-1 dimension is a no-op:</p>
<pre><code class="lang-auto">UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.HYBRID_SHARD since the world size is 1.
</code></pre>
<p>However, the fallback seems to set the <em>world size to 1</em>, which results in gradients <strong>not</strong> being synchronized across replicated ranks. This causes model parameters to diverge between GPUs.</p>
<p>To verify this, I ran a test on a single node with 2 GPUs and used a replicate dimension of 2. After one training step, I compared model parameters and optimizer states across ranks:</p>
<ul>
<li><img src="https://discuss.pytorch.org/images/emoji/apple/white_check_mark.png?v=14" title=":white_check_mark:" class="emoji" alt=":white_check_mark:" loading="lazy" width="20" height="20"> With DDP and FSDP + <code>NO_SHARD</code>: parameters were consistent.</li>
<li><img src="https://discuss.pytorch.org/images/emoji/apple/cross_mark.png?v=14" title=":cross_mark:" class="emoji" alt=":cross_mark:" loading="lazy" width="20" height="20"> With FSDP + <code>HYBRID_SHARD</code> (which falls back to <code>NO_SHARD</code>): parameters diverged.</li>
</ul>
<p>It seems like the fallback isn’t preserving the DDP-like behavior as expected. I’m not sure if this is a bug or intentional, but we were hoping to use a single FSDP script to test different mesh configurations (including <code>(n, 1)</code>), and this issue breaks that workflow.</p>
<p>Would appreciate any clarification on whether this is expected. Thanks!</p>

username: H-Huang
cooked: <p>This looks like a bug. Can you file an issue on the PyTorch repository with an example script to reproduce it? <a href="https://github.com/pytorch/pytorch/issues" class="inline-onebox" rel="noopener nofollow ugc">GitHub · Where software is built</a></p>
CUDA not detected on Windows

username: Hector
cooked: <p>Apologies for a simple question. I have installed Torch on Windows 10.</p>
<p>Torch has Cuda .<br>
print( torch.<strong>version</strong>)<br>
2.7.0+cu118</p>
<p>Cuda also seems to be available on my computer.<br>
nvcc --version<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2022 NVIDIA Corporation<br>
Built on Tue_May__3_19:00:59_Pacific_Daylight_Time_2022<br>
Cuda compilation tools, release 11.7, V11.7.64<br>
Build cuda_11.7.r11.7/compiler.31294372_0</p>
<p>But:<br>
print(torch.cuda.is_available() )<br>
False</p>
<p>This is a new install of both Conda and Torch.</p>
<p>Any help would be appreciated.</p>

username: dbrenes
cooked: <p>Hello <a class="mention" href="/u/hector">@Hector</a></p>
<p>This error could be due to your NVIDIA GPU not being able to support the CUDA 11.8 version, since you installed PyTorch with CUDA 11.8 support (<code>2.7.0+cu118</code>).</p>
<p>Perhaps you can check which CUDA version does your GPU support, by matching your GPU’s compute capability, microarchitecture and supported CUDA SDK using <a href="https://en.wikipedia.org/wiki/CUDA#GPUs_supported" rel="noopener nofollow ugc">these tables</a>.</p>
<hr>
<p><em>Machine Learning Engineer at <a href="https://www.ridgerun.ai/" rel="noopener nofollow ugc">RidgeRun.ai</a></em><br>
<em>Contact us: <a href="mailto:support@ridgerun.ai">support@ridgerun.ai</a></em></p>

username: Hector
cooked: <p>Thank you, Daniela. This helps.</p>