akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Why is torch.rand different on different devices for big indices

username: Jackmin801
cooked: <p>Hey PyTorch team. Big fan of PyTorch! Great work <img src="https://discuss.pytorch.org/images/emoji/apple/fire.png?v=14" title=":fire:" class="emoji" alt=":fire:" loading="lazy" width="20" height="20">.</p>
<p>Im having trouble understanding when <code>torch.rand</code> is the same across devices.<br>
I ran the code below on a 3090, A100 and H100:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import numpy as np
import random

torch.use_deterministic_algorithms(True)
torch.manual_seed(0)
torch.cuda.manual_seed(0)
np.random.seed(0)
random.seed(0)

a = torch.rand(1_000_000, device='cuda', generator=torch.Generator(device='cuda').manual_seed(10))

print(a[32 * 48 * 82 - 2: 32 * 48 * 82 + 2])
print(a[32 * 48 * 144 - 2: 32 * 48 * 144 + 2])
</code></pre>
<p>3090 output</p>
<pre><code class="lang-auto">tensor([0.5327, 0.4037, 0.5954, 0.2597], device='cuda:0')
tensor([0.6247, 0.3700, 0.1930, 0.9421], device='cuda:0')
</code></pre>
<p>A100 output</p>
<pre><code class="lang-auto">tensor([0.5327, 0.4037, 0.4867, 0.9688], device='cuda:0')
tensor([0.1685, 0.8799, 0.5954, 0.2597], device='cuda:0')
</code></pre>
<p>H100 output</p>
<pre><code class="lang-auto">tensor([0.5327, 0.4037, 0.4867, 0.9688], device='cuda:0')
tensor([0.1685, 0.8799, 0.7291, 0.4280], device='cuda:0')
</code></pre>
<p>They all obtain the same result up till index <code>125952</code>. The A100 and H100 start differing at <code>221184</code>.<br>
Is this behavior intended?</p>
<p>Is the only way for me to obtain the same result across devices for bigger tensor sizes to do smaller rands and cat the result?</p>

username: ptrblck
cooked: <p>Yes, this behavior is expected as we do not guarantee to create reproducible results on different setups. If you want to reuse the same pseudorandom values you could serialize them and load in the application which should use them.</p>
Autograd about CUDAExtension

username: XinyuFu
cooked: <p>I write a forward function by using CUDAExtension kernal in device cuda. I want to use the autograd model to compute the gradient of this processing. Which document can help me to complete this work?</p>

username: ptrblck
cooked: <p>Assuming you are using your custom extension in an <code>autograd.Function</code> you would need to implement the <code>backward</code> method manually as well. PyTorch will handle the backward automatically if you use native and differentiable PyTorch ops.</p>

username: XinyuFu
cooked: <p>Thank you for your help, Sir!<br>
But I am confused in this part. If I write the backward function by CUDAExtension kernal,where can autograd be demonstrated? Are the backward function and the gradient equivalent？Perhaps there is something wrong with my understanding.Does this mean that if I use the custom cuda extension function, I cannot take advantage of automatic differentiation?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="XinyuFu" data-post="3" data-topic="219960">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/e95f7d/48.png" class="avatar"> XinyuFu:</div>
<blockquote>
<p>Are the backward function and the gradient equivalent？</p>
</blockquote>
</aside>
<p>The <code>backward</code> function computes the gradients for all inputs as seen in e.g. <a href="https://docs.pytorch.org/docs/stable/notes/extending.html#example">this tutorial</a>.</p>
<aside class="quote no-group" data-username="XinyuFu" data-post="3" data-topic="219960">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/x/e95f7d/48.png" class="avatar"> XinyuFu:</div>
<blockquote>
<p>Does this mean that if I use the custom cuda extension function, I cannot take advantage of automatic differentiation?</p>
</blockquote>
</aside>
<p>Yes, as PyTorch won’t be able to reason about the custom operations you are using unless you use PyTorch operations.</p>

username: XinyuFu
cooked: <p>Thank you for you help, Sir! Your remark answer my doubts.</p>
"no-op" (?!) set_default_device("mps") makes optimizer creation crash

username: lebigot
cooked: <p>Using <code>set_default_device()</code> in the following code makes it raise an exception when creating an optimizer, <em>despite looking like a no-op</em>:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch import nn

# 1. Ensure MPS is available
assert torch.backends.mps.is_available(), "MPS device not found"

# Commenting the following out fixes the optimizer creation!
torch.set_default_device("mps")

# 2. Set device
device = torch.device("mps")

# 3. Define a minimal model
model = nn.Linear(10, 1)

# 4. Move model to MPS before creating the optimizer
model = model.to(device)

# 5. Create Adam optimizer *after* moving model to device
optimizer = torch.optim.Adam(model.parameters())
</code></pre>
<p>This raises <code>RuntimeError: Placeholder storage has not been allocated on MPS device!</code>, but everything works fine without the <code>set_default_device()</code>. Is this expected, or is this a bug?</p>
<p>Getting an answer to this would be useful, because <code>set_default_device()</code> is convenient, as it seems to avoid doing many <code>.to(device)</code>!</p>
<h1><a name="p-470435-some-details-on-the-configuration-1" class="anchor" href="#p-470435-some-details-on-the-configuration-1"></a>Some details on the configuration</h1>
<p>PyTorch 2.7.0 (conda)<br>
Python 3.12</p>
Why `max_memory_allocated()` exceeds physical GPU memory size?

username: GS_H
cooked: <p>I’m training a 14B model using FSDP. By monitoring the <code>torch.cuda.max_memory_allocated()</code>, I found that it exceeds my GPU’s maximal memory by around 60GB.</p>
<p>I’m wonding why this happen? Does pytorch use some sort of virtual memory management? If so, is there any documentation?</p>
<p>Thanks!</p>

username: albanD
cooked: <p>Hey!</p>
<p>I does not do that. The only case I’ve seen this happen is when running on Windows machines where the OS is automagically doing “unified memory like” feature that we cannot control in any ways.</p>

username: GS_H
cooked: <p>Thanks for the reply! I’m currently using a linux machine (ubuntu 24.02) with CUDA 12.8. As far as I know, the unified memory seems not present in my system.</p>
<p>I plan to investigate the source code.</p>

username: GS_H
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Have you ever seen this phenomenon before? I’d appreciate if you provide any reference to the related topics <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Both training and validation loss not reducing

username: Soham_Bhaumik
cooked: <p>I am trying to fine-tune a transformer/encoder based pose estimation model available on huggingface</p>
<p>When passing “labels” attribute to the forward pass of the model, the model returns “Training not enabled”. The core logic I have implemented is as follows. Since the model outputs heatmaps, I use a post-processing pipeline to get back the keypoint predictions in the image space, and compute a MSE Loss between these reconstructed keypoint (using soft argmax) and ground truth keypoints.</p>
<p>Is this a correct way of thinking? Comparing heatmap to heatmap might seem more intuitive, but I didn’t want to write the keypoint to heatmap and add the specific image processor’s normalization with the worry that they might go off-scale Things I have tried:</p>
<p>modified the model’s heatmap head to predict for 24 keypoints for dogs instead of the 17 for humans it was trained on 2.added a Simple Adapter network right after the layer norm and before the model’s heatmap head unfreeze some of the backbone layer’s gradually track loss with both normalized and unnormalized keypoints. Added gradient clipping. the post processing pipeline is a differentiable approxiamtion of <a href="https://github.com/huggingface/transformers/blob/main/src/transformers/models/vitpose/image_processing_vitpose.py" class="inline-onebox" rel="noopener nofollow ugc">transformers/src/transformers/models/vitpose/image_processing_vitpose.py at main · huggingface/transformers · GitHub</a> Tuning LRs However, gradient flow through the head and the adapter and the deeper encoder layers have been very very small.</p>
<p>here is the notebook link: <a href="https://github.com/sohamb91/ML/blob/main/ViTpose_update_loss.ipynb" class="inline-onebox" rel="noopener nofollow ugc">ML/ViTpose_update_loss.ipynb at main · sohamb91/ML · GitHub</a></p>
<p>Looking forward to any discussions/help.</p>
<p>I’m at a complete “loss”.</p>
<p>Edit: i have tried normalising the keypoint coordinates wrt the bounding boxes as well as compare the heat map output of the model with Gaussians generated at the label keypoint coordinates</p>
Get_backend() returns undefined even when NCCL is available

username: joshkim
cooked: <p>Hi,</p>
<p>I’m currently using torchrun to do distributed training.</p>
<p>I use <code>init_process_group()</code> without any parameters in my code since it can automatically pick the backend based on the device type of my current environment:  <a href="https://docs.pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 2.7 documentation</a></p>
<p>My current environment has GPUs and I can see it being properly used with NCCL based on the NCCL logs generated by the NCCL_DEBUG env var.</p>
<p><strong>Problem</strong>: However, <code>torch.distributed.get_backend()</code> still returns ‘undefined’ even when the lazy initialization is done and the use of NCCL is so clear in my code. I also tried to manually put the default process group as its input parameter but it still returns the same. It only returns <code>nccl</code> when I explicitly set <code>init_process_group('nccl')</code>.</p>
<p><strong>Question</strong>: Is this an expected behavior? How can I detect which backend torch is using if I use <code>init_process_group</code> with no parameter?</p>

username: ptrblck
cooked: <p>I guess this is expected as the backend was undefined during the PG creation and I don’t see where it’s being updated (<a href="https://github.com/pytorch/pytorch/blob/e06a08059a378ee48a3e33b413c4c0763661aae0/torch/distributed/distributed_c10d.py#L236">code</a>). CC <a class="mention" href="/u/kwen2501">@kwen2501</a> in case the backend is supposed to be updated after the init call selects the default one for the corresponding device.</p>

username: kwen2501
cooked: <p>Yes, that’s expected – at init, no backend was tied to the Process Group in your case.<br>
The collective works because PyTorch will try to find a backend for the device identified by your tensor.<br>
The no-parameter init is more for a case where your program need to perform collectives on multiple device types, e.g. “cuda” and “cpu”. Thus, when <code>get_backend()</code> is queried, it does not know one because it is “floating”.</p>

username: joshkim
cooked: <p>Thanks for the explanation, <a class="mention" href="/u/kwen2501">@kwen2501</a> and <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
Retrieving Attention Weights from scaled_dot_product_attention

username: anto1481
cooked: <p>Hi all!<br>
I was computing flash attention in my model implementation and I was just wondering if there is any way of getting the attention weights that are computed in <code>torch.nn.functional.scaled_dot_product_attention</code>. From the framework itself, it does not seem to be any possibility.<br>
I wondered also if I could just implement this function in my model (implementation is defined in <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.scaled_dot_product_attention.html" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.functional.scaled_dot_product_attention — PyTorch 2.2 documentation</a>) and just use the <code>sdp_kernel</code> and get the same results (which will enable me to retrieve the attention weights from the function).</p>
<p>Thank you!</p>

username: Miquel_Espinosa
cooked: <p><a class="mention" href="/u/anto1481">@anto1481</a> Did you manage to do it? How did you end up retrieving the attention weights?<br>
Thanks.</p>

username: b10901187
cooked: <p><a class="mention" href="/u/miquel_espinosa">@Miquel_Espinosa</a> <a class="mention" href="/u/anto1481">@anto1481</a> Did you manage to do it? I also face this problem but don’t know how to solve it…</p>

username: Miquel_Espinosa
cooked: <p>I think I ended up doing something like this…</p>
<p><code>attention_scores = torch.einsum('bhqd,bhkd-&gt;bhqk', q, k)</code><br>
<code>self.attn_weights = F.softmax(attention_scores / math.sqrt(q.size(-1)), dim=-1)</code></p>

username: b10901187
cooked: <p>What I current facing is that if I just do the torch.enisum or torch.dot will cause OOM, I think is the F.scaled_dot_product_attention will do flash attention or something to optimize the memory</p>

username: tom
cooked: <p>A large part of the memory (and time) savings of flash attention is avoiding to materialize the attention matrix.<br>
Some things you can do (if you have the time):</p>
<ul>
<li>be sure to cast q and k to small precision before the einsum and maybe detach.</li>
<li>You could also do the computation head by head.</li>
<li>Instead of doing the softmax, you could
<ul>
<li>Compute the <code>log_sum_exp</code> and subtract the value inplace (so the logsumexp is 0, i.e. you have done <code>log_softmax</code> inplace,</li>
<li>compute the <code>exp</code> inplace</li>
</ul>
</li>
</ul>

username: b10901187
cooked: <p>what I currently want is the query @ key before the softmax，so is there a best way to get it?</p>

username: Menki_Guo
cooked: <p>One method I used is that I am actually not interested to the whole NxN attention map, and I am only interested in kxN where k is the number of interested tokens. Extract them first will save memory and time a lot if k &lt;&lt; N</p>
New to Contribution

username: Achintya_Paningapall
cooked: <p>Hi there!<br>
I am new to contributions and just submitted my first PR. I was wondering if there was a slack or a communication channel for me to learn more. Additionally, are there meetings that I can join to understand how I can be a good developer for PyTorch?</p>

username: ptrblck
cooked: <p>Congrats on your first contribution! If you are interested in a specific topic I would recommend just asking here. If you want to discuss specific feature requests or PRs GitHub would be a better place.</p>

username: Achintya_Paningapall
cooked: <p>Thanks for the response! How would you recommend I learn more about the community? Additionally, how should I work so that I can work towards becoming an Issue Triager?</p>

username: ptrblck
cooked: <p>I would recommend sticking around and contributing to the community in any way you like. E.g. you could read through forum posts here and help others in using PyTorch (which is a great opportunity for you to learn more about PyTorch as well) or by fixing issues, proposing features, etc. on GitHub. I don’t think there is a magic recipe besides continuously contributing to the stack.</p>

username: Achintya_Paningapall
cooked: <p>Thanks for the response! How do I get involved with introducing new features?</p>
PyTorch memory usage keeps increasing

username: Morac
cooked: <p>I’m running <a href="https://github.com/brandon929/FramePack" class="inline-onebox" rel="noopener nofollow ugc">GitHub - brandon929/FramePack: FramePack for macOS and Apple Silicon</a> (FramePack video generation) on my M4 Mac Studio 128 GB ram using the nightly PyTorch libraries and while it works, the amount of memory used is not only huge, but keeps growing after every video generation.  It starts out using around 85 GB per video and then each subsequent one uses another 20 GB or so until the point where my Mac needs to start swapping and the memory pressure goes into the yellow.</p>
<p>Basically I have to kill the server after every use or I run out of memory.</p>
<p>I asked the author and he said he’s not seeing that.  He says it stays at a fairly stable 75 GB.  Is there some kind of debugging I can enable to see where all the memory is going?</p>
Autograd FLOP Calculation with Higher Order Derivatives

username: alexi
cooked: <p>I’m working on FLOP (floating-point operation) calculation and ran into a challenge with higher-order derivatives when using <code>torch.autograd</code>. Specifically, I’m looking to compute FLOPs for operations that involve <code>autograd.grad(..., create_graph=True)</code> to support higher-order derivative calculations.</p>
<p>I’ve noticed that none of the existing FLOP profiling libraries I’ve tested (like <code>torch.profiler</code>, <code>fvcore</code>, or <code>ptflops</code>) seem to support FLOP calculation when higher-order derivatives are involved. They either don’t account for autograd operations (and just estimate it using the forward pass) or fail when <code>create_graph=True</code> is enabled.</p>
<p>In theory, the computational cost of higher-order derivatives could scale as O(n^2), where n is the number of parameters, but in practice, I’ve observed that it often runs much faster. I’m sure there are optimizations pytorch makes under the hood to avoid to avoid naive calculation of the hessian. Has anyone else explored FLOP profiling for such use cases or can share insights on how autograd handles these operations under the hood to make it faster than the theoretical worst-case? Or, if necessary, is it possible to add support for these types of FLOP calculations?</p>
<p>Thanks</p>

username: alexi
cooked: <p>I just wanted to bump this and see if anyone knows. Thanks</p>

username: soulitzer
cooked: <p>I don’t think there are advanced optimizations for higher-order derivatives, but simply doing .grad twice should just compute the hvp rather than the full hessian.</p>
<p>One way to estimate flops is to use</p>
<pre><code class="lang-auto">from torch.utils.flop_counter import FlopCounterMode
</code></pre>
<p>Although, I’m not sure how many ops it supports right now.</p>

username: alexi
cooked: <p>Unfortunately this doesn’t support higher order derivatives. Is it known the number of FLOPs the HVP takes compared to normal first order backpropagation?</p>
Intel® Arc™ 140T GPU support machine learning acceleration?

username: Dengjun21
cooked: <p>My GPU is Intel® Arc™ 140T GPU, window11 system, can I support machine learning acceleration? If yes, is there any documentation on how to do it?</p>

username: paulge
cooked: <p>Should be supported as of October 24 <a href="https://pytorch.org/blog/intel-gpu-support-pytorch-2-5/" rel="noopener nofollow ugc">https://pytorch.org/blog/intel-gpu-support-pytorch-2-5/</a>. Here is the “Getting started” guide <a href="https://pytorch.org/docs/main/notes/get_start_xpu.html" rel="noopener nofollow ugc">https://pytorch.org/docs/main/notes/get_start_xpu.html</a></p>

username: Dengjun21
cooked: <p>Thansk you reply, I would try to do that with url which your provided，If I finished sucessful I will update response in this topic. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=12" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>

username: python
cooked: <p>I recommend checking out the Intel oneAPI documentation, especially the sections related to the Deep Learning Toolkit. It should provide you with detailed instructions on setup and usage. Additionally, you might find resources on integrating with popular frameworks like TensorFlow and PyTorch.</p>
[Question] How to solve no kernel image is available for execution on the device?

username: LiaoLW
cooked: <p>Hello, I got an error with:<br>
“scores = torch.matmul(xq, xk.transpose(2, 3)) / math.sqrt(self.head_dim)<br>
RuntimeError: CUDA error: no kernel image is available for execution on the device<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.”<br>
I’m using A100 with torch=2.4.0_cu118.<br>
However, I have tested directly performing torch.matmul with two torch.rand().cuda() and get no error, what is the problem?</p>

username: ptrblck
cooked: <p>Your code tries to call a kernel which was not compiled for your device. Since the <code>matmul</code> itself works (and your PyTorch binary supports Ampere devices) you could check which other 3rd party library you are using which might be compiled in a wrong way.</p>
Feedback on AMD/Intel ARC GPUs compared to Nvidia

username: Darklinux
cooked: <p>Hello to the community: simple question: do you feel a drop in performance with Intel or AMD GPUs compared to Nvidia architectures, do you have any examples? Thank you</p>

username: KFrank
cooked: <p>Hi Dark!</p>
<aside class="quote no-group" data-username="Darklinux" data-post="1" data-topic="219866" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/darklinux/48/76836_2.png" class="avatar"> Darklinux:</div>
<blockquote>
<p>performance with Intel or AMD GPUs compared to Nvidia architectures, do you have any examples?</p>
</blockquote>
</aside>
<p>I have one data point from a laptop that has both a cpu with integrated intel arc graphics<br>
and a discrete nvidia chip.  (Note, the comparison is really going to depend on your specific<br>
hardware and if you’re interested high-end performance, a laptop would hardly be the most<br>
the most relevant platform.)</p>
<p>See this post:</p>
<aside class="quote quote-modified" data-post="1" data-topic="218200">
  <div class="title">
    <div class="quote-controls"></div>
    <img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/timings-for-intel-arc-graphics-xpu-vs-nvidia-rtx-3000-gpu-on-a-laptop/218200">Timings for intel arc graphics "xpu" vs. nvidia rtx 3000 gpu (on a laptop)</a> 
  </div>
  <blockquote>
    Hi All! 
For your delectation: 
Short story:  The intel “xpu” is about half as fast as the nvidia gpu and is about six times 
faster than running on the cpu. 
Pytorch has recently started supporting intel gpus (on a prototype basis).  See 
<a href="https://pytorch.org/docs/main/notes/get_start_xpu.html" rel="noopener nofollow ugc">“Getting Started on Intel GPU.”</a> 
Specific hardware: thinkpad p16v gen2 with intel arc graphics integrated into the 
intel core ultra 9 185h main processor and a separate nvidia rtx 3000 ada generation 
laptop gpu. 
The timing test consists of simple model with…
  </blockquote>
</aside>

<p>Best.</p>
<p>K. Frank</p>

username: Darklinux
cooked: <p>Thank you for this quick response</p>
PyTorch 2.x causes divergence during training with mixed precision

username: TitusPullo
cooked: <p>I was previously using PyTorch 1.13. I have a regular mixed precision setup where I use autocast. There are noticeable speed ups with mixed precision enabled, so everything works fine.</p>
<p>However, I need to update my PyTorch version to 2.5+. When I do this, my training losses start increasing a lot around 25000 iterations. Disabling mixed precision resolved the issue, but I need it for training speed.  I tried 2.5 and 2.6. Same issue happens with both.</p>
<p>My model contains transformers.</p>
<p>I tried using bf16 instead of fp16, it started diverging even earlier (around 8000 iterations).</p>
<p>I am using GradScaler, and I logged its scaling factor. When using fp16, It goes as high as 1 million, and quickly reduces to 4096 when divergence happens. When using bf16, scale keeps increasing even after divergence happens.</p>
<p>I also have the following, not sure if it can cause this issue:</p>
<blockquote>
<pre><code>    torch.backends.cudnn.enabled = True 
    torch.backends.cudnn.benchmark = True 
    torch.autograd.set_detect_anomaly(False) 
    torch.autograd.profiler.profile(False) 
    torch.autograd.profiler.emit_nvtx(False) 
</code></pre>
</blockquote>
<p>What might be the issue? I can provide further details if necessary. Thanks in advance.</p>

username: TitusPullo
cooked: <p>I tried everything at this point. I am not sure what causes this. Any help would be appreciated.</p>
How can I visualize this NN?

username: cpanuccio
cooked: <p>class Net(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.fc1 = nn.Linear(X_train.shape[1], 16)<br>
self.dropout = nn.Dropout(0.5)<br>
self.fc2 = nn.Linear(16, 1)</p>
<pre><code>def forward(self, x):
    x = torch.relu(self.fc1(x))
    x = self.dropout(x)
    x = self.fc2(x)
    return x
</code></pre>
<p>model = Net()</p>

username: cpanuccio
cooked: <p><span class="mention">@ptrblck_de</span> need your help!</p>

username: Eduardo_Lawson
cooked: <p>Do you want to see like a summary of the layers?</p>

username: cpanuccio
cooked: <p>yes and/or input, activation and output nodes with connections if possible</p>

username: Eduardo_Lawson
cooked: <p>For a summary of the layers with input and outpu you can go for [<a href="https://pypi.org/project/torchinfo/" class="inline-onebox" rel="noopener nofollow ugc">torchinfo · PyPI</a>].</p>
<p>For visualization of the blocks itself, you can go for SummaryWriter and use writer.add_graph.</p>

username: cpanuccio
cooked: <p>can you paste the exact lines of code I need here? I keep getting the RuntimeError: mat1 and mat2 shapes cannot be multiplied error</p>

username: Eduardo_Lawson
cooked: <p>I supposed a random input:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torchinfo import summary


import numpy as np
X_train = np.random.rand(100, 10)  # example: 100 samples, 10 features

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(X_train.shape[1], 16)
        self.dropout = nn.Dropout(0.5)
        self.fc2 = nn.Linear(16, 1)
    
    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.fc2(x)
        return x

model = Net()

# Creation of a  dummy input
batch_size = 32
input_features = X_train.shape[1]
dummy_input = torch.randn(batch_size, input_features)

model_summary = summary(model, 
                        input_size=(batch_size, input_features),
                        col_names=["input_size", "output_size", "num_params", "kernel_size", "mult_adds"],
                        verbose=2)
print(model_summary)
</code></pre>
Just a silly question. Whats the difference between import torch.nn as nn vs from torch import nn?

username: torch_newbie
cooked: <p>My linter is always complaining about <code>import torch.nn as nn</code> and tells me to change it to <code>from torch import nn</code>. Is there a difference between the 2?</p>

username: ptrblck
cooked: <p>Purist might see a difference (and please post these). I would use whichever approach you like. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=14" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
Tensorboard add_graph (actually jit.trace) does not work with None inputs

username: turbotimon
cooked: <p>Hi, I have a problem where a forward method takes optional inputs:</p>
<pre data-code-wrap="python"><code class="lang-python">def forward(self, x_in):
    x1, x2 = x_in
   if x2 is None:
     ...
</code></pre>
<p>so when I want to <code>add_graph</code> for a case without the x2 provided…</p>
<pre data-code-wrap="python"><code class="lang-python">writer.add_graph(model, input_to_model=((x, None)))
</code></pre>
<p>… I get an error:</p>
<pre data-code-wrap="sh"><code class="lang-sh">"**/jit/_trace.py", line 1278, in trace_module
    module._c._create_method_from_trace(
RuntimeError: Type 'Tuple[Tensor, NoneType]' cannot be traced. Only Tensors and (possibly nested) Lists, Dicts, and Tuples of Tensors can be traced
</code></pre>
<p>Is there an easy workaround?</p>
<ul>
<li>changing the signature of <code>forward</code> is not an option</li>
<li>in this simple example, also providing x2 would not hurt, but the real example is more complex, resulting in another graph</li>
</ul>
<p>Thanks in advance</p>
<hr>
<p>(Minimal reproducible example on request available)</p>

username: turbotimon
cooked: <p>btw: I found out there is an open request for something similar, but its open since a while… <a href="https://github.com/pytorch/pytorch/issues/18850" class="inline-onebox" rel="noopener nofollow ugc">Allow tracing of models which output `None` · Issue #18850 · pytorch/pytorch · GitHub</a></p>
<p>the only solution I came up with, is either wrapping or monkey-patching the forward method of the model to not take a tuple but optional arguments…</p>
Rewriting this without using libtorch

username: L_R
cooked: <p>I want to rewrite this without using libtorch</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">  torch::Tensor pred_masks = torch::nn::functional::interpolate(
      masks.index({scores_mask, torch::indexing::Ellipsis}),
      torch::nn::functional::InterpolateFuncOptions().size(
          std::vector&lt;int64_t&gt;({input_height_, input_width_})));
</code></pre>
<p>in which <code>masks</code> and <code>scores_mask</code> are both tensors.<br>
I don’t want to use libtorch because the library introduces great space cost (more than 4GB for the GPU version) to my project, but I dont know how to rewrite the functions such as the <code>torch::nn::functional::InterpolateFuncOptions()</code> and .<code>index</code> by using pointers like <code>float *</code><br>
anyone has ideas about it?</p>
The pytorch script get stuck and exit without any error

username: Smith_Jack
cooked: <p>Hello. I am training my model on a single GPU (on a remote sever) and the script gets stuck and corrupts randomly without any error output. When I print the size of some tensors in the loss function, it seems that the scripts runs normally. GPT-4o says I can add <code>torch.cuda.synchronize()</code> after the cuda operations and add <code>flush=True</code> in the <code>print</code> function, but it does not help. Besides, if I run the script with <code>mprof run</code>, the script can also run normally. The script just like a Schrodinger Cat, if I try to observe it then the corruption does not occur.<br>
How can I find out what’s wrong with my script.</p>
<p>PS: When the script gets stuck, the SSH client exits automatically and I cannot connect it again in a short time.</p>

username: albanD
cooked: <p>Hey!</p>
<p>What you’re describing sounds like a machine that runs our of RAM. And so it hangs for a while trying to recover and start killing process in a hard way.<br>
Can you check that RAM usage is ok on your machine?</p>

username: Smith_Jack
cooked: <p>Thanks. I have solved this problem. As you said, it is indeed a hardware error instead of the problem of pytorch.</p>
Does flex_attention still require a sequence length of 128 or not?

username: divinho
cooked: <p>This requirement was specified in the original blogpost: <a href="https://pytorch.org/blog/flexattention/" rel="noopener nofollow ugc">https://pytorch.org/blog/flexattention/</a></p>
<p>“FlexAttention requires that all sequence lengths be a multiple of 128 – this will be addressed soon.”</p>
<p>It is not mentioned in the docs in either torch version 2.5/2.7 though: <a href="https://docs.pytorch.org/docs/2.5/nn.attention.flex_attention.html#torch.nn.attention.flex_attention.flex_attention" class="inline-onebox" rel="noopener nofollow ugc">torch.nn.attention.flex_attention — PyTorch 2.5 documentation</a></p>
<p>What’s the current status? I did not see a mention of it in the recent torch releases.</p>
<p><a class="mention" href="/u/chillee">@Chillee</a></p>

username: Chillee
cooked: <p>This limitation has been addressed since about 2.6 I think.</p>
Can I build and install PyTorch and LibTorch together only using CMake?

username: DapengFeng
cooked: <p>I want to use CMake to build PyTorch and LibTorch instead of using python setup.py install</p>

username: python
cooked: <p>You can build and install both PyTorch and LibTorch with CMake. First, clone the repository using <code>git clone --recursive https://github.com/pytorch/pytorch</code>, then install the necessary dependencies. Create a build directory and configure with <code>cmake ..</code>, adding any options you need (like CPU-only or GPU). Build with <code>cmake --build . --config Release</code>, and finally, install using <code>sudo cmake --install .</code></p>
Is my training fast or not?

username: Bjorn_Lindqvist
cooked: <p>I’m training AlexNet on the CIFAR dataset on a really beefy HPC node equipped with 256 cores and 8 AMD Instinct MI250X GPUs. Time per epoch is about five seconds. According to calflops one forward+backward pass (input is resized to 227x277) takes 4.3 GFLOPS. There are 50k images in the training set so it comes out to roughly 43 TFLOPS/s.</p>
<p>Is this FAST? Given the hardware I have at my disposal should I be satisfied? I have to train lots of models with different hyper parameter settings from scratch so time I invest in improving the training setup is well-worth it.</p>
Where the grades from backward graph will be copied to param.grade in cuda graphs

username: Tiru_B
cooked: <p>Hi all</p>
<p>I am currently working with torch.compile cuda graphs,  i see for simple MLP layer 3 graphs ,forwad ,backward and loss graphs are separately compiled and data being shared across the graphs. One observation is backward pass output tensors i,e gradients have different memory location that what optimizer is being recived ,also optimizer updated weights are explicitly coped in runtime wrapper.py to parameter tensors.</p>
<p>So i want to know where exactly backwards graph outputs are copied to parameter.grads ,and why its being done.</p>
<p>Based on this will implement similar thing for our own accelerator</p>
<p><a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/marksaroufim">@marksaroufim</a></p>
CTC loss inputs and input lengths

username: alicemabille
cooked: <p>Hello,<br>
I’m using CTC loss for ASR, for phoneme recognition. My model is a pretrained Whisper tiny encoder (not frozen during training) and a linear classification head as a decoder, with a LogSoftmax acitvation layer. My dataset is L2-Arctic.<br>
I have looked everywhere for the exact thing to use as inputs for pytorch’s CTCLoss, but it was not very clearly explained anywhere. I can’t find the torch.ctc_loss() function source code, only the wrappers.<br>
I am currently using sequences of classes deduced from the max predicted probability for each window, with blank padding (class n°0). For the input lengths, I am taking these inputs and deleting repetitions as well and blanks.</p>
<ul>
<li>Should I input something else ?</li>
<li>Should the input lengths take repetitions into account ?</li>
<li>Should the input lengths take blanks into account ?</li>
</ul>
<p>As for the targets, I am inputing classes without the blanks as well:</p>
<pre><code class="lang-auto">target_lengths = torch.count_nonzero(batch["phonemes"], axis=1)
</code></pre>
<p>I use the <code>zero_infinity=True</code> flag because after some time my model predicts so much silence that the input lengths are shorter than targets.</p>
<p>I am asking this because my model learns to predict too much silence, and the wrong classes. Using the full predictions lengths without removing blanks nor repetitions made the model learn to predict the same class for every window. (ex: predicting 2222222222222222222222222222)</p>
Gradient of a mixed network's output with respect to ONE tensor

username: Sherlock_Holmes
cooked: <p>Suppose I have an MLP: R^d → R. Since MLP(x_i) does not depend on x_j for j != i, I can retrieve the gradient by  simply calling <code>grad(output.sum(), x)</code>.</p>
<p>However, I am now in the case where my network is a GNN, so the above trick doesn’t work anymore (because the computation for each <code>x_i</code> depends on <code>x_j</code> for all <code>j</code>). Again, my output is a real number for each <code>x_i</code>, and I want to compute its gradient with respect to <code>x_i</code>.</p>
<p>Any ideas? Thanks!</p>

username: soulitzer
cooked: <p>Hmm I feel like I don’t completely understand. torch.autograd.grad(output.sum(), x) would compute the gradient of the output wrt x regardless of the dependency structure of the function. Or are you saying the inputs depend on one another?</p>

username: Sherlock_Holmes
cooked: <p>The case in which that trick works is the following grad_xi (sum_j fj(xj) ) = grad_xi fi(xi), but in my case f is fi = fi(x1, x2, …).</p>
Question about Slurm and PyTorch

username: Stev1
cooked: <p>I want to run 4 PyTorch scripts on a compute node with 4 GPUs in parallel. If I tell Slurm to use one GPU per script(I submit 4 jobs using one GPU each and let them run on one node in parallel), is it enough to just set device = “cuda” in the scripts? Does PyTorch automatically know which GPU to use or do I have to specify which of them to use?</p>

username: ptrblck
cooked: <p>If you make a single GPU visible via e.g. <code>CUDA_VISIBLE_DEVICES</code> or via e.g. a <code>docker</code> command you can use <code>device="cuda"</code> inside your script.</p>

username: Bjorn_Lindqvist
cooked: <p>Set <code>CUDA_VISIBLE_DEVICES=0</code> for the first script, <code>CUDA_VISIBLE_DEVICES=1</code>, for the second script, and so on. Otherwise all four scripts will use the same GPU.</p>
Any suggestion about long time serie prediction as this kind?

username: WEIKE_LIAO
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1a304af49cd44f086ffdc0202ca4fbde587747e8.png" data-download-href="https://discuss.pytorch.org/uploads/default/1a304af49cd44f086ffdc0202ca4fbde587747e8" title="1fd5e64e6cbc76984cc906dc559f6fc"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a304af49cd44f086ffdc0202ca4fbde587747e8_2_690x376.png" alt="1fd5e64e6cbc76984cc906dc559f6fc" data-base62-sha1="3JFSCP7in9l6Re9NkS5xwjydTde" width="690" height="376" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a304af49cd44f086ffdc0202ca4fbde587747e8_2_690x376.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a304af49cd44f086ffdc0202ca4fbde587747e8_2_1035x564.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a304af49cd44f086ffdc0202ca4fbde587747e8_2_1380x752.png 2x" data-dominant-color="EEF6FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1fd5e64e6cbc76984cc906dc559f6fc</span><span class="informations">2386×1303 418 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Such loooooooooong and tense curves, in my case with 250,000 to 75,000 data points. Im trying to do the prediction with transformer o time domain+frequency domain…</p>
<p>If you have any idea, you r very welcome to share it!!!</p>
About running time of Convolution

username: Stark_Ivan
cooked: <p>According to the discussion on PyTorch forums ([<a href="https://discuss.pytorch.org/t/convtranspose1d-extremely-slow-on-gpu-t4-slower-than-cpu/88977" class="inline-onebox">ConvTranspose1d extremely slow on GPU (T4), slower than CPU</a>]), I understand that the initial iteration of GPU operations can be slow. I am training a model with convolutional layers using input images of random resolutions. In each epoch, I select images corresponding to approximately 48 different resolutions to be sampled. Consequently, the convolution operations are extremely slow at the beginning of each epoch. Do you have any advice on how to mitigate this?</p>
Mismatch in data shapes

username: saphira_ELARA
cooked: <p>As I’m trying to run a training loop and training model on batches of data, I get this error message “ValueError: Expected input batch_size (16) to match target batch_size (32).” I</p>
<p>Below is my code :</p>
<p><span class="hashtag-raw">#import</span> tqdm for progress bar<br>
from tqdm import tqdm<br>
from tqdm.auto import tqdm</p>
<p><span class="hashtag-raw">#set</span> the seed and start the timer<br>
torch.manual_seed(42)<br>
train_time_start_on_cpu = timer()</p>
<p><span class="hashtag-raw">#set</span> the number of epochs<br>
epochs = 10</p>
<p><span class="hashtag-raw">#create</span> a training and test loop<br>
for epoch in tqdm(range(epochs)):<br>
print(f"Epoch: {epoch}\n------")</p>
<pre><code>#training
train_loss = 0
#add a loop to loop through the training batches
for batch, (X,y) in enumerate(train_dataloader):
    m_0.train()

    #forward pass
    y_pred = m_0(X)

    #calculate the loss per batch
    loss = loss_f(y_pred, y)
    train_loss += loss #to accumulate the train loss

    #optimize the zero grad
    optim.zero_grad()

    #loss backward // backpropagation step
    loss.backward()

    #optimizer step 
    optim.step()

    #print out what's happening 
    if batch %400 == 0:
        print(f" Looked at {batch * len(X)}/{len(train_dataloader.dataset)} samples")

#divide total train loss by lenght of train dataloader
train_loss /= len(train_dataloader)

#testing 
test_loss, test_acc = 0, 0
m_0.eval()
with torch.inference_mode():
    for X_test, y_test in test_dataloader:

        #forward pass
        test_pred = m_0(X_test)

        #calculate the loss accumulatively
        test_loss = loss_f(test_pred, y_pred)

        #calculate the accuracy
        test_acc = accuracy_fn(y_true=y_test, y_pred=test_pred.argmax(dim=1))

    #calculate the test loss average per batch
    test_loss /= len(test_dataloader)

    #calculate the test acc average per batch
    test_acc /= len(test_dataloader)

#print out what's happening 
print(f"Train loss : {train_loss:.4f} | Test loss : {test_loss:.4f} | Test acc : {test_acc:.4f}")
</code></pre>
<p><span class="hashtag-raw">#calculate</span> the training time<br>
train_time_end_on_cpu = timer()<br>
total_train_time_m_0 = print_train_time(start=train_time_start_on_cpu,<br>
end=train_time_end_on_cpu,<br>
device=str(next(m_0.parameters()).device)<br>
)</p>
<p>Could someone please help me ?</p>
How to assignment between two parameters with their string names?

username: ljppro
cooked: <p>Hi,<br>
I’m working on dynamically changing the parameter values. For example, I have a linear layer with parameter weight and bias, and I need to change their values in runtime.<br>
Below works for me:</p>
<ol>
<li>create a new parameter in the model <strong>init</strong>:<br>
name = “weight1”<br>
np = nn.Parameter(data=torch.randn(self.weight.data.shape), requires_grad=True)<br>
self.register_parameters(name, np)</li>
<li>and I found that self.weight1 is created by Pytorch and visible in forward() and I can directly replace self.weight with self.weight1 as below:<br>
self.weight = self.weight1</li>
</ol>
<p>But as I have many parameters in this model, it’s tedious to do it in hard-coding way, so I do it dynamically:<br>
for n, p in self.param_dict.items():<br>
# n = “weight”<br>
name = n + str(1) #  name = “weight1”<br>
# get the reference to original “self.weight”<br>
pn = self.get_parameter(n) # fc.weight<br>
# get the reference to the new parameter “fc.weight1”<br>
pnp =  self.get_parameter(name)</p>
<pre><code>        # set fc.weight1 to fc.weight
        pn = pnp
</code></pre>
<p>I expected pn = pnp would work in the same way as “self.weight = self.weight1”, but it didn’t. It seems that pnp will directly point to the parameter itself, instead of self.weight, so that assignment doesn’t work and in forward() the parameter value of self.fc.weight doesn’t change at all.<br>
Do we have a way to return the variable fc.weight instead of the parameter fc.weight pointed to? Or how can we do the assignment dynamically by their names “weight” and “weight1”, instead of the assignment between hardcoding variable names for example, fc.weight=fc.weight1?</p>
<p>Many thanks!</p>

username: ljppro
cooked: <p>I can change weight by pn.data = pnp.data, but it doesn’t work for me because autograd won’t work here.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="ljppro" data-post="1" data-topic="219811">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/43a26b/48.png" class="avatar"> ljppro:</div>
<blockquote>
<p>Or how can we do the assignment dynamically by their names “weight” and “weight1”, instead of the assignment between hardcoding variable names for example, fc.weight=fc.weight1?</p>
</blockquote>
</aside>
<p>I don’t think any assignment would be a good idea since you would also need to update e.g. the parameters passed to an optimizer. Assuming you want to change the actual values of the trainable parameter, copy the new data in a <code>no_grad</code> context into the already registered parameter inplace:</p>
<pre data-code-wrap="python"><code class="lang-python">with torch.no_grad():
    self.weight1.copy_(new_data)
</code></pre>

username: ljppro
cooked: <p>Many Thanks! I will have a try on this <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Transformer Loss remains same

username: Mafaz03
cooked: <p>I am trying to implement ImageBind model but a smaller version of it (consisting of text, image and video modalities only)</p>
<p>Although the implementation looks solid,<br>
Source: <a href="https://github.com/facebookresearch/ImageBind/" class="inline-onebox" rel="noopener nofollow ugc">GitHub - facebookresearch/ImageBind: ImageBind One Embedding Space to Bind Them All</a></p>
<p>the loss while training is very constant. ie the loss stays at Loge(Batch size)<br>
if batch size is 3, the loss stays at 1.098</p>
<p>here is a snippet:</p>
<pre><code class="lang-auto">import torch
from torch.nn.functional import normalize, cross_entropy
from tqdm import tqdm

DEVICE = "cpu"

def contrastive_loss(x, y, temperature=0.07):
    """
    InfoNCE-style contrastive loss between x and y.
    """
    logits = (x @ y.T) / temperature
    labels = torch.arange(x.size(0)).to(x.device)
    loss_i2t = cross_entropy(logits, labels)
    loss_t2i = cross_entropy(logits.T, labels)
    return (loss_i2t + loss_t2i) / 2

optimizer = torch.optim.AdamW(model.parameters(), lr=5e-3, weight_decay=0.01)
loss_fn = nn.MSELoss()

# Training loop
for epoch in range(EPOCHS):
    model.train()
    total_loss = 0.0
    step = 0

    pbar = tqdm(image_dl, desc=f"Epoch {epoch+1}/{EPOCHS}")

    for image, text in pbar:
        # Move data to device
        image = image.to(DEVICE)
        text = text.to(DEVICE)

        # Forward pass
        inputs = {"vision": image, "text": text}
        embeddings = model(inputs)

        # Normalize embeddings
        img_emb = normalize(embeddings["vision"], dim=-1)
        txt_emb = normalize(embeddings["text"], dim=-1)

        # Compute contrastive loss
        loss = contrastive_loss(img_emb, txt_emb)
        print(loss)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # Logging
        total_loss += loss.item()
        step += 1
        pbar.set_postfix(loss=total_loss / step)

    print(f"Epoch {epoch+1}: Avg Loss = {total_loss / step:.4f}")
</code></pre>
<p>please view my implementation: <a href="https://github.com/Mafaz03/ImageBind" class="inline-onebox" rel="noopener nofollow ugc">GitHub - Mafaz03/ImageBind</a><br>
extra points :</p>
<ol>
<li>requires_grad is true for all</li>
<li>param.grad.abs().mean() is almost 0 for all</li>
<li>I have also tried all initialization such as Xavier and that didnt fix it.</li>
</ol>
<p>Thank you.</p>