akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Run quantized model on GPU

username: Eline
cooked: <p>Hi<br>
I want to run inference on a quantized model using GPU, but it only works on CPU.</p>
<p>I have quantized a pytorch nn model using quantize_dynamic_jit and torch.jit.trace. It performs int8 quantization on the linear layers. It has reduced the size of the model with approximately 71% and it is still very accurate. The problem is I only seem to be able to run the inference using CPU and not GPU, so the original model still outperforms the quantized. Thus right now im only using CPU to quantize and for inference. I believe its only possible to use the CPU for quantization though.</p>
<p>I have also tried to load the model as a PyTorch NN module instead of a TorchScript, but it seems the model architecture is changed.<br>
Any help is greatly appreciated</p>

username: marksaroufim
cooked: <p>Our GPU quantization support is in ao <a href="https://github.com/pytorch-labs/ao" class="inline-onebox">GitHub - pytorch-labs/ao: torchao: PyTorch Architecture Optimization (AO). A repository to host AO techniques and performant kernels that work with PyTorch.</a> - repo is still under heavy development but please feel free to ping me if you run into any issues</p>

username: Gadam_Setty_Vedapani
cooked: <p>I am also trying to run the dynamic quantized model (which i saved as .pt file) and load the model on the GPU with the below code but the Kernel is dying immediately…<br>
Any Help regarding this will be very helpful</p>
<p>Code:</p>
<p>import time<br>
from transformers import AutoModelForSeq2SeqLM,AutoTokenizer<br>
import torch</p>
<p>bit8_model=AutoModelForSeq2SeqLM.from_pretrained(“google/flan-t5-large”)<br>
quantized_model_path = “./quantized_model_full.pt”<br>
device = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)<br>
bit8_model = bit8_model.load_state_dict(torch.load(quantized_model_path,map_location=device))<br>
bit8_model.to(device)<br>
bit8_model.eval()</p>
Aot_export_module captures more forward variables/parameters

username: sycz00
cooked: <p>Hey folks,</p>
<p>I want to capture the forward and backward computation of the swin-transformer.<br>
The code below tries to do that and I encountered the following issue: the number of parameters of the model is 173 while the number of variables decomposed in the forward function is 186 - 2 (1 for target and 1 for inputs).<br>
The discrepancy was reproduced by other visual transformers as well and it scaled with the <strong>depths</strong> parameter of swin-transformer. Could you help me somehow ?</p>
<pre><code class="lang-auto">import torchvision
from torch._functorch.aot_autograd import aot_export_module
from torch._functorch.partitioners import default_partition
import torch.fx as fx
import torch
import torch.nn as nn

num_classes = 1000

user_defined_model = torchvision.models.swin_transformer.SwinTransformer(patch_size= [4,4], embed_dim=96, depths=[2, 2, 6, 2], num_heads=[3, 6, 8, 12], window_size=[7,7],num_classes=num_classes).cuda()

class WrappedModel(nn.Module):
    def __init__(self,nn_model,num_classes):
        super(WrappedModel, self).__init__()
        self.model = nn_model
        self.num_classes = num_classes

    def forward(self, x, target):
        output = self.model(x)
       
        target_onehot_float = torch.nn.functional.one_hot(target,self.num_classes).float()
        loss = nn.CrossEntropyLoss()(output.float(),target_onehot_float)  
        
        
        return (loss, output.detach())

counter = 0
for name, para in user_defined_model.named_parameters():
    if para.requires_grad:
        counter += 1
print(f"Number model parameters {counter}") #173
    
inputs = torch.randn(size=(1,3,224,224), requires_grad=True).cuda()
target = torch.randint(high=num_classes,size=(1,),dtype=torch.long).cuda()

model = WrappedModel(user_defined_model,num_classes)
m, _ = aot_export_module(model, [inputs,target], trace_joint=True, \
                             output_loss_index=0, decompositions=None)
fwd, bwd = default_partition(m, [inputs, target], num_fwd_outputs=1)
fwd.to_folder(folder="forward_vit",module_name="Forward")
bwd.to_folder(folder="backward_vit",module_name="Backward")
</code></pre>
<p>Libraries: Conda env.<br>
-pytorch-2.4.0-py3.8_cuda11.8_cudnn9.1.0_0<br>
-nvcc: Build cuda_11.8.r11.8/compiler.31833905_0<br>
-python 3.8</p>
<p>I also tried with the latest PyTorch 2.5</p>

username: bdhirsh
cooked: <p>There are also some buffers in the model. if I include this code in your script:</p>
<pre><code class="lang-auto">for name, para in user_defined_model.named_buffers():
     buffer_counter += 1
</code></pre>
<p>I get:</p>
<pre><code class="lang-auto">Number model parameters 173
Number model buffers 12
</code></pre>

username: sycz00
cooked: <p>Hey</p>
<p>yes you are right, they might come from PyTorch register_buffer() function. I found a way to compile it properly but had to remove some essential parts of the architecture. I think there’s more work to do.</p>
Quantization of depthwise 1d convolution with QAT is slower than non-quantized

username: hongxuzhu
cooked: <p>I am a rookie in QAT and try to decrease the latency of a speech recognition model with QAT(from float32 to qint8). After experiment, I find QAT do improve the processing speed of conv1d and conv2d layers a lot. However, when quantize the depthwise 1d convolution layers, I found that this layer is even slower after quantization. Do I use a wrong qconfig or something? Any suggestions for this? Thank you!<br>
Details:<br>
depthwise 1d convolution layer:<br>
(depthwise_conv): Conv1d(512, 512, kernel_size=(31,), stride=(1,), padding=(15,), groups=512)<br>
Qconfig during QAT training(‘x86’):<br>
(activation_post_process): HistogramObserver(min_val=inf, max_val=-inf)<br>
processing time before QAT: 4.7ms<br>
processing time after QAT: 56ms</p>

username: Vasiliy_Kuznetsov
cooked: <p>The performance impact of quantization is dependent on your problem size, hardware, config settings - any of those things could be preventing a speedup on a specific layer.</p>

username: hongxuzhu
cooked: <p>Thank you for your reply. As all the convolution layers are faster than before, but depthwise layers are even 10 times slower. I am just wondering if this is normal, or I use a wrong config on it. Do you have any suggestion on the qconfig for the depthwise convolution?</p>
DataParallel multi-gpu RuntimeError: chunk expects at least a 1-dimensional tensor

username: anubhgar
cooked: <p>Hi, I am trying to run my model on multiple gpus using DataParallel by setting <code>model = nn.DataParallel(model).cuda()</code>, but everytime getting this error - RuntimeError: chunk expects at least a 1-dimensional tensor (chunk at /pytorch/aten/src/ATen/native/TensorShape.cpp:184).</p>

username: ptrblck
cooked: <p>Could you check the shape of your input data?<br>
<code>nn.DataParallel</code> splits the input tensor in dim0 and sends each chunk to the specified GPUs.<br>
It seems you are trying to pass a 0-dimensional tensor?</p>
Unrecognized CMAKE_SYSTEM_NAME = Windows while building Pytorch in Windows

username: Jenet
cooked: <p>Hi,</p>
<p>I am trying to build Pytorch v2.5.1 from source in Windows with a latest clang compiler.<br>
Steps followed:</p>
<ol>
<li>git clone --recursive <a href="https://github.com/pytorch/pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a></li>
<li>cd pytorch</li>
<li>Git checkout v2.5.1</li>
<li>git submodule sync</li>
<li>git submodule update --init --recursive</li>
<li>python setup.py install</li>
</ol>
<p>But I am facing errors as:</p>
<pre><code class="lang-auto">CMake Error at aten/src/ATen/native/quantized/cpu/qnnpack/CMakeLists.txt:59 (message):
  Unrecognized CMAKE_SYSTEM_PROCESSOR = AMD64


-- Configuring incomplete, errors occurred!
</code></pre>
During evaluation process, my model keeps predicting the same result

username: L_Z
cooked: <p>I am training a CNN+LSTM model for binary classification with dropout layer (0.3 dropping ratio). And I am using nn.CrossEntropyLoss along with optimizer as Adam. I can see my output values keep changing during my model.train() process, but during my model.eval(), the output value keeps repeating for different data, like exactly the same value. For example tensor([0.9600, -1.2629]), no better what data I feed into. How to solve this issue? Also, I tried to disable the dropout layer, but still the outputs are the same throughout. Thanks in advance.</p>
Reduce memory usage during iterative algorithm

username: sherlock.h
cooked: <p>Hello!</p>
<p>This is a snippet of my code (<code>create_graph</code> is required in my use case, and <code>energy</code> is a big CNN)</p>
<pre data-code-wrap="python"><code class="lang-python">for _ in range(n_steps):
  sample_energy = energy.forward(sample)
  sample_score = grad(sample_energy.sum(), sample, create_graph=True)[0]
  value, sample = f(value, sample, sample_energy, sample_score)

loss = g(value)
loss.backward()
optimizer.step()
optimizer.zero_grad()
</code></pre>
<p>What are my options to reduce the memory footprint of these model + grad calls?</p>
<p><code>f</code> and <code>g</code> perform common, uninteresting <code>Tensor</code> operations like multiplication, summation, etc …</p>

username: Soumya_Kundu
cooked: <p>Potentially, you could try gradient clipping since you explicitly mention its a large model. Not sure what sort of dips that will cause performance metric wise.</p>
<p>Assuming your training follows this structure in general, you could also look into setting <code>loss.backward(retain_graph=False)</code> .</p>
F.scaled_dot_product_attention get query @ key

username: b10901187
cooked: <p>Hello, everyone, I want to know how to get the query @ key in F.scaled_dot_product_attention, I use the below code but still got OOM, I can use the F.scaled_dot_product attention and don’t get the OOM, plz help…<br>
or how to change the source code of F.scaled_dot_product_attention</p>
<pre><code class="lang-auto">def chunk_dot_product(query, key, num_chunks=2000):
    # query, key shape: [batch_size, num_heads, seq_len, head_dim]
    batch_size, num_heads, seq_len, head_dim = query.shape
    chunk_size = seq_len // num_chunks
    
    # 初始化輸出張量列表
    attn_chunks = []
    
    for i in range(num_chunks):
        chunk_weights = []
        # 取出當前 query chunk: [batch_size, num_heads, chunk_size, head_dim]
        q_chunk = query[:, :, i*chunk_size:(i+1)*chunk_size]
        
        # 對 key 也進行分塊處理
        for j in range(num_chunks):
            k_chunk = key[:, :, j*chunk_size:(j+1)*chunk_size]
            
            # 計算部分注意力權重
            # [batch_size, num_heads, chunk_size, chunk_size]
            chunk_attn = torch.matmul(q_chunk, k_chunk.transpose(-1, -2))
            chunk_weights.append(chunk_attn)
            
            # 適時清理記憶體
            if j &lt; num_chunks - 1:  # 最後一個 chunk 不需要清理
                del k_chunk
                torch.cuda.empty_cache()
        
        # 在序列長度維度上連接: [batch_size, num_heads, chunk_size, seq_len]
        row_weights = torch.cat(chunk_weights, dim=-1)
        attn_chunks.append(row_weights)
        
        # 清除中間結果
        del chunk_weights
        del q_chunk
        torch.cuda.empty_cache()
    
    # 最後將所有塊組合起來: [batch_size, num_heads, seq_len, seq_len]
    attn_weight = torch.cat(attn_chunks, dim=2)
    return attn_weight

# Efficient implementation equivalent to the following:
def scaled_dot_product_attention(query, key, value, attn_mask=None, dropout_p=0.0, is_causal=False, scale=None) -&gt; torch.Tensor:
    L, S = query.size(-2), key.size(-2)
    scale_factor = 1 / math.sqrt(query.size(-1)) if scale is None else scale
    attn_bias = torch.zeros(L, S, dtype=query.dtype).to(query.device)
    if is_causal:
        assert attn_mask is None
        temp_mask = torch.ones(L, S, dtype=torch.bool).tril(diagonal=0)
        attn_bias.masked_fill_(temp_mask.logical_not(), float("-inf"))
        attn_bias.to(query.dtype)

    if attn_mask is not None:
        if attn_mask.dtype == torch.bool:
            attn_bias.masked_fill_(attn_mask.logical_not(), float("-inf"))
        else:
            attn_bias += attn_mask
    attn_weight = chunk_dot_product(query, key) * scale_factor
    attn_weight += attn_bias
    # after the lora masking do the softmax
    # attn_weight = torch.softmax(attn_weight, dim=-1)
    # attn_weight = torch.dropout(attn_weight, dropout_p, train=True)
    
    return attn_weight
</code></pre>
MultiHeadAttention Activation Initialization

username: JFernando4
cooked: <p>In the current version, in the <code>MultiheadAttention</code> activation in <code>torch.nn.modules.activation</code>, in the <code>_reset_parameters</code> method in <a href="https://github.com/pytorch/pytorch/blob/f2d7fe12d816f351d696b6ba02083cbef4b9f157/torch/nn/modules/activation.py#L1116C9-L1116C26" rel="noopener nofollow ugc">line 1116</a>, all the parameters of the activations are initialized using Xavier initialization except for the output projection weight. The output projection weight is missing and is likely being initialized using the default initialization for linear layers. This is the method:</p>
<pre><code class="lang-auto">def _reset_parameters(self):
        if self._qkv_same_embed_dim:
            xavier_uniform_(self.in_proj_weight)
        else:
            xavier_uniform_(self.q_proj_weight)
            xavier_uniform_(self.k_proj_weight)
            xavier_uniform_(self.v_proj_weight)

        if self.in_proj_bias is not None:
            constant_(self.in_proj_bias, 0.0)
            constant_(self.out_proj.bias, 0.0)
        if self.bias_k is not None:
            xavier_normal_(self.bias_k)
        if self.bias_v is not None:
            xavier_normal_(self.bias_v)
</code></pre>
<p>Since <code>self.out_proj.weight</code> is missing in this function, it is initialized as: <code>torch.nn.init.kaiming_uniform_(self.out_proj.weight, a=math.sqrt(5))</code> inside of the linear layer  <code>reset_parameters</code> method <a href="https://github.com/pytorch/pytorch/blob/1ce533867f325e8d4b41646f8dd3ddd7f6ea15fe/torch/nn/modules/linear.py#L114" rel="noopener nofollow ugc">here</a>.</p>
<p>I was wondering if this was an intentional choice or if it’s a bug.</p>
TensorRT: Errors in PTQ Example

username: Tim_Langer
cooked: <p>I was trying to follow the TensorRT PTQ examples here to implement INT8 quantization using TensorRT:<br>
<a href="https://pytorch.org/TensorRT/ts/ptq.html#how-to-create-your-own-ptq-application-in-python" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/TensorRT/ts/ptq.html#how-to-create-your-own-ptq-application-in-python</a></p>
<p>However, if I try to use torch_tensorrt.ptq.DataLoaderCalibrator() and torch_tensorrt.ptq.CalibrationAlgo.ENTROPY_CALIBRATION_2, I get following error message: AttributeError: module ‘torch_tensorrt’ has no attribute ‘ptq’.</p>
<p>I found in another place that the ptq module was available under ts (seems to be up-to-date w.r.t. the Github repo), so I also tried:<br>
torch_tensorrt.ts.ptq.DataLoaderCalibrator()<br>
and torch_tensorrt.ts.CalibrationAlgo.ENTROPY_CALIBRATION_2,<br>
but the result was the same:<br>
AttributeError: module ‘torch_tensorrt.ts’ has no attribute ‘ptq’</p>
<p>I am using following versions under Python 3.11:</p>
<ul>
<li>tensorrt 10.3.0</li>
<li>torch 2.5.1</li>
<li>torch_tensorrt 2.5.0</li>
<li>nvidia-modelopt 0.21.1</li>
</ul>
<p>I think the examples given under <a href="https://pytorch.org/TensorRT/ts/ptq.html#how-to-create-your-own-ptq-application-in-python" class="inline-onebox" rel="noopener nofollow ugc">Post Training Quantization (PTQ) — Torch-TensorRT v2.6.0.dev0+69c83d4 documentation</a> don’t refer to the current module structure.<br>
Can you give me a hint here and possibly update the example?</p>
<p>Cheers,<br>
Tim</p>

username: ptrblck
cooked: <p>CC <a class="mention" href="/u/narendasan">@narendasan</a> for TorchTRT</p>

username: narendasan
cooked: <p>The quantization API from TensorRT has been deprecated so this API doesn’t really work anymore and will be removed soon. You can use model-opt + the dynamo frontend to do this now. <a href="https://pytorch.org/TensorRT/tutorials/_rendered_examples/dynamo/vgg16_ptq.html" class="inline-onebox" rel="noopener nofollow ugc">Deploy Quantized Models using Torch-TensorRT — Torch-TensorRT v2.6.0.dev0+69c83d4 documentation</a></p>

username: Tim_Langer
cooked: <p>Hi Naren, thanks a lot - the using model-opt &amp; dynamo frontend works for me!</p>
Understanding the traceback of a compiler error with cache size limit

username: Johannes_Vogt
cooked: <p>When I compile my custom nn.module, I do not get an error. However when I use the model, I get</p>
<pre><code class="lang-auto">W0121 11:29:04.514000 140682235495680 torch/_dynamo/convert_frame.py:357] torch._dynamo hit config.cache_size_limit (8)
W0121 11:29:04.514000 140682235495680 torch/_dynamo/convert_frame.py:357] function: 'torch_dynamo_resume_in_forward_at_1213' (/path_to_my_module/module.py:1213)
W0121 11:29:04.514000 140682235495680 torch/_dynamo/convert_frame.py:357] last reason: L['___stack0'] == 1737455341.032532
W0121 11:29:04.514000 140682235495680 torch/_dynamo/convert_frame.py:357] To log all recompilation reasons, use TORCH_LOGS="recompiles".
</code></pre>
<p>How do I tackle this problem, e.g. what are the principles to set the cache_size_limit? Is it a problem when I set it to e.g. 64 with torch._dynamo.config.cache_size_limit = 64?</p>
<p>Or is the problem actually somewhere else?</p>
<p>Where do I set TORCH_LOGS= ‘recompiles’? And does this error message mean, that it recompiles every time?</p>
Targeted Image Augmentation

username: MUSTAFA_CAGRI_CIVICI
cooked: <p>Hello everyone,</p>
<p>I am working on the model that will be trained on 4 channels medical data. I noticed that labels of the images are not distributed fairly. In order to fetch the importance of the low-weighted class more, I am planning to augment the train data. But if i apply generic image augmentation, it might be no effect because the augmentation will effect the same on each class. I need to augment especially low weighted class so that the model would see them a lot. I can tune it manually like low classes will be augmented 5 times while high class will be 2 times. But it is too risky for the model to be manipulated.<br>
Are there any methods or advices that you have or you like using? It would be very appreciated for me.</p>

username: KFrank
cooked: <p>Hi Mustafa!</p>
<aside class="quote no-group quote-modified" data-username="MUSTAFA_CAGRI_CIVICI" data-post="1" data-topic="215621" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mustafa_cagri_civici/48/74964_2.png" class="avatar"> MUSTAFA_CAGRI_CIVICI:</div>
<blockquote>
<p>I need to augment especially low weighted class so that the model would see them a lot.<br>
…<br>
Are there any methods or advices that you have or you like using?</p>
</blockquote>
</aside>
<p>I don’t see anything wrong with augmenting classes that appear less<br>
frequently more heavily than classes that appear more frequently.</p>
<p>But my guess is that you would be a little bit better off using a<br>
<a href="https://pytorch.org/docs/2.5/data.html#torch.utils.data.WeightedRandomSampler" rel="noopener nofollow ugc">WeightedRandomSampler</a> to sample the less-frequent classes more<br>
heavily (and if you do augment your training data, don’t augment any<br>
differently based on class).</p>
<p>(You could also use the <code>weight</code> constructor argument for<br>
<code>CrossEntropyLoss</code>, if this is a multi-class classification problem,<br>
or the conceptually similar <code>pos_weight</code> constructor argument for<br>
<code>BCEWithLogitsLoss</code>, if this is a binary classification problem.  But<br>
I would prefer the <code>WeightedRandomSampler</code> approach unless it is<br>
likely that any given batch would have duplicate samples in it.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: MUSTAFA_CAGRI_CIVICI
cooked: <p>Many thanks <a class="mention" href="/u/kfrank">@KFrank</a>, I will work on it now. Thanks for sharing your advice.</p>
Efficiently Training Multiple Large Models with PyTorch FSDP: Best Practices?

username: erfan_mhi
cooked: <p>Hi everyone,</p>
<p>I’m working on a reinforcement learning framework for training large language models (LLMs) and need to run three very large models simultaneously. The challenge is that these models are too big to fit into a single GPU, and I’m exploring ways to distribute them effectively across GPUs and nodes using PyTorch’s Fully Sharded Data Parallel (FSDP).</p>
<p>Here are my specific questions:</p>
<ol>
<li>Can PyTorch FSDP handle sharing a single GPU among these three models?</li>
<li>If not, what would be the best approach to distribute the models across GPUs/nodes while ensuring efficient memory usage and communication?</li>
</ol>
Where to find <gloo/algorithm.h> header file in libtorch

username: tttk
cooked: <p>I want to use gloo to do collective comminication in torch cpu tensor by libtorch c++ api。I find demo in pytorch unit test source code: test/cpp/c10d/ProcessGroupGlooTest.cpp. It require include “torch/csrc/distributed/c10d/ProcessGroupGloo.hpp”, ProcessGroupGloo.hpp is one of libtorch header file. But ProcessGroupGloo.hpp require gloo header file, eg: &lt;gloo/algorithm.h&gt;. Where can I find gloo header file, It not in libtorch include folder.</p>
RuntimeError: Given groups=1, weight of size [512, 1536, 3, 3], expected input[1, 1024, 32, 32] to have 1536 channels, but got 1024 channels instead

username: jungns7234
cooked: <p>I encountered this error while running UNet. My code is as follows. What is the issue? My input image is 256 x 256.</p>
<p>from unet_parts import *</p>
<p>class UNet(nn.Module):<br>
def <strong>init</strong>(self, n_channels=1, n_classes=1, bilinear=False):<br>
super(UNet, self).<strong>init</strong>()<br>
self.n_channels = n_channels<br>
self.n_classes = n_classes<br>
self.bilinear = bilinear</p>
<pre><code>    self.inc = (DoubleConv(n_channels, 64))
    self.down1 = (Down(64, 128))
    self.down2 = (Down(128, 256))
    self.down3 = (Down(256, 512))
    factor = 2 if bilinear else 1
    self.down4 = (Down(512, 1024 // factor))
    self.up1 = (Up(1024, 512 // factor, bilinear))
    self.up2 = (Up(512, 256 // factor, bilinear))
    self.up3 = (Up(256, 128 // factor, bilinear))
    self.up4 = (Up(128, 64, bilinear))
    self.outc1 = (OutConv(64, n_classes))
    self.outc2 = (OutConv(64, n_classes))
    self.outc3 = (OutConv(64, n_classes))
    self.outc4 = (OutConv(64, n_classes))


def forward(self, x):
    x1 = self.inc(x)
    x2 = self.down1(x1)
    x3 = self.down2(x2)
    x4 = self.down3(x3)
    x5 = self.down4(x4)
    x = self.up1(x5, x4)
    x = self.up2(x, x3)
    x = self.up3(x, x2)
    x = self.up4(x, x1)
    fake_bin1 = self.outc1(x)
    fake_bin2 = self.outc2(x)
    fake_bin3 = self.outc3(x)
    fake_bin4 = self.outc4(x)

    return fake_bin1, fake_bin2, fake_bin3, fake_bin4

def use_checkpointing(self):
    self.inc = torch.utils.checkpoint(self.inc)
    self.down1 = torch.utils.checkpoint(self.down1)
    self.down2 = torch.utils.checkpoint(self.down2)
    self.down3 = torch.utils.checkpoint(self.down3)
    self.down4 = torch.utils.checkpoint(self.down4)
    self.up1 = torch.utils.checkpoint(self.up1)
    self.up2 = torch.utils.checkpoint(self.up2)
    self.up3 = torch.utils.checkpoint(self.up3)
    self.up4 = torch.utils.checkpoint(self.up4)
    self.outc1 = torch.utils.checkpoint(self.outc1)
    self.outc2 = torch.utils.checkpoint(self.outc2)
    self.outc3 = torch.utils.checkpoint(self.outc3)
    self.outc4 = torch.utils.checkpoint(self.outc4)
</code></pre>

username: Sam_d
cooked: <p>Add channel 1 to your input image. your input should be (1, 256, 256) if  your image is grayscale and if you are working in PyTorch.</p>
Bring tensors to GPU in a batch

username: SzymonOzog
cooked: <p>Iterating over tensors and calling <code>.to("cuda")</code> causes a lot of overhead while the GPU is not really busy. Is there a way to tell pytorch to bring a batch of tensors to the GPU?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/3/43bbd4d0f77d429d3e9f912ef161700f49d7f8ed.png" data-download-href="https://discuss.pytorch.org/uploads/default/43bbd4d0f77d429d3e9f912ef161700f49d7f8ed" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/43bbd4d0f77d429d3e9f912ef161700f49d7f8ed_2_203x500.png" alt="image" data-base62-sha1="9FcpRVbiOucbx6K1WIEYiLLJ52R" width="203" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/3/43bbd4d0f77d429d3e9f912ef161700f49d7f8ed_2_203x500.png, https://discuss.pytorch.org/uploads/default/original/3X/4/3/43bbd4d0f77d429d3e9f912ef161700f49d7f8ed.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/3/43bbd4d0f77d429d3e9f912ef161700f49d7f8ed.png 2x" data-dominant-color="D6D2DF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">243×597 12.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: JuanFMontesinos
cooked: <p>I think you could use tensordicts, but I’m unsure how allocation on GPU happens under the hood.<br>
<a href="https://pytorch.org/tensordict/main/overview.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tensordict/main/overview.html</a></p>

username: vmoens
cooked: <p>Yes tensordict will execute that somewhat faster using non_blocking data transfer. Happy to look at the profile if you’re willing to share it with that lib!</p>
Switching from "model.eval()" to "model.train()" severely degrades performance

username: cloudydory
cooked: <p>Most related issues I can find on the Internet are that model performance degrades when switching from “model.train()” to “model.eval()”. However, I had exactly the opposite problem: model performance degrades when switching from “model.eval()” to “model.train()”.</p>
<p>I am training a UNet for a segmentation task on 3 GPUs using PyTorch DDP. The only modules that behave differently in training and evaluation mode are “SyncBatchNorm” layers in each convolution block. I met this problem when trying to resume training from the saved model and optimizer state dicts. The loss are very high in the first few steps (though it quickly goes down later). This happens even when I set the initial learning rate to a very low value.</p>
<p>I isolate the problem using the following code:</p>
<pre><code class="lang-auto">with torch.no_grad(), torch.autocast("cuda", dtype=torch.bfloat16, enabled=True):
    model.eval()
    mask_pred_batch = model(image_batch)
    loss = compute_loss(mask_pred_batch, mask_gt_batch)  # calculate loss
    if local_rank == 0:
        print(loss.item())

    model.train()
    mask_pred_batch = model(image_batch)
    loss = compute_loss(mask_pred_batch, mask_gt_batch)  # calculate loss
    if local_rank == 0:
        print(loss.item())
</code></pre>
<p>And the printed loss values are:</p>
<pre><code class="lang-auto">0.10580593347549438
17.66183853149414
</code></pre>
<p>Essentially, the same batch is passed into the model twice, one under evaluation mode and the other under training mode. The batch comes from the same training data in the previous training session, so the first loss value is very low. However, just changing the model to training mode dramatically increase the loss to the level of an untrained model.</p>
<p>The image size is 512x512, the batch size per GPU is 16 (so in total 48 since I use SyncBatchNorm and have 3 GPUs). The same batch size is used before and after resuming. The BatchNorm2D’s momentum is 0.1 (the default value). The PyTorch version is 2.5.1.</p>
<p>I cannot understand why this happens. Hope someone can give me some hints and help to solve the problem.</p>
Why my Traing accuracy remains constant

username: arsh_sharma
cooked: <pre><code class="lang-auto">glove_model = api.load("glove-wiki-gigaword-100") 

class TextDataset(Dataset):
    def __init__(self, data):
        self.data = data
        self.X = self.data["text"]
        self.y = self.data["label"]
        
    def __len__(self):
        return len(self.data)
    
    def __getitem__(self, idx):
        text = self.data['text'][idx]
        label = self.data['label'][idx]

        text_embeddings = [self.get_embedding(word) for word in text]
        text_embeddings = torch.tensor(text_embeddings,dtype=torch.float32)
        return text_embeddings, label

    def get_embedding(self,word):
        if word in glove_model:
            return glove_model[word]
        return np.zeros(100)

train_dataset = TextDataset(train_data)
val_dataset = TextDataset(val_data)
test_dataset = TextDataset(test_data)

# create dataloaders
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=32, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=32, shuffle=True)
</code></pre>
<p>and anfter this I feed it to LSTM directly as shown below</p>
<pre><code class="lang-auto">epochs = 10
# Train the model

for epoch in range(epochs):
    train_accuracy = 0
    for i, (texts, labels) in enumerate(train_loader):
        texts = texts.to(device)
        labels = labels.to(device)
        
        # Forward pass
        outputs = LSTM(texts)
        loss = criterion(outputs, labels)
        
        # Backward pass
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # Accuracy
        _, predicted = torch.max(outputs, dim=1)
        train_accuracy += (predicted == labels).sum().item()
        
    
    print (f'Epoch {epoch+1},Loss: {loss.item()}, Accuracy: {train_accuracy/len(train_data)}')
</code></pre>

username: ptrblck
cooked: <p>Is this a double post from <a href="https://discuss.pytorch.org/t/my-accuracy-remains-constant/215397">here</a> or what is the difference between these topics?</p>

username: vdw
cooked: <p>What is <code>LSTM</code> in your training loop. Do you create a new <code>torch.nn.LSTM</code> in each loop? If so, this won’t work. You have to create it once, e.g.:</p>
<pre><code class="lang-auto">lstm = torch.nn.LSTM(...)
...
for ...
    for ....
        outputs = lstm(texts)
</code></pre>
<p>In any case, <code>torch.nn.LSTM</code> outputs a nested tuple <code>(output, (h_n, c_n))</code>. Not sure how you can directly give this to <code>criterion</code>.</p>
Problem in Backpropagation through a sample in Beta distribution in pytorch

username: Jimut123
cooked: <p>Say I have obtained some <strong>alphas</strong> and <strong>betas</strong> as parameters from a neural network which will be parameters of the Beta distribution. Now, I sample from the Beta distribution and then calculate some loss and back-propagate via the samples obtained. Is it possible to do that? given that after the sampling process I do <strong>.requires_grad_(True)</strong> to the sample and then compute the loss? This surely works, but it looks like the loss is not converging, is there any other way to do this in Pytorch?</p>
<p>Say, I get the following variables via some neural network:</p>
<pre data-code-wrap="python"><code class="lang-python">mu, sigma, pred = model.forward(input)
</code></pre>
<p>Where say, <strong>mu</strong> is the (batch_size x 30) shaped tensor, similarly <strong>sigma</strong> is (batch_size x 30) shaped tensor. I compute the <strong>alphas</strong> and <strong>betas</strong> using the <strong>mu</strong> and <strong>sigma</strong> obtained from a Neural Network (both of the same shape (batch_size x 30)), and then sample it via a beta distribution as follows:</p>
<pre><code class="lang-auto">def sample_from_beta_distribution(alpha, beta, eps=1e-6):
	# Clamp alpha and beta to be positive
	alpha_positive = torch.clamp(alpha, min=eps)
	beta_positive = torch.clamp(beta, min=eps)
    
	# Create a Beta distribution
	# This will automatically broadcast to handle the batch dimension
	beta_dist = torch.distributions.beta.Beta(alpha_positive, beta_positive)
    
	# Sample from the distribution
	# This will return samples of shape [38, 30]
	samples = beta_dist.sample()
    
	return samples
</code></pre>
<p>Here, I take the <strong>samples</strong> which is of the same shape as (batch_size x 30), perform some operations on it, and then calculate the loss. I expected the gradient to propagate through this, but looks like the loss is not converging.</p>
<p>Any leads would help. Please note, this is not as simple as the reparameterization trick in the standard Normal distribution.</p>

username: Jimut123
cooked: <p>Looks like <code>.rsample()</code> is doing the trick here, which is keeping the computational graph alive…</p>
Loss.backward() called after torch.nograd()

username: hanxix
cooked: <p>Hi, i’m working my way through the Dive Into Deep Learning book and found the following code (pasted below or linked <a href="https://d2l.ai/chapter_linear-regression/linear-regression-scratch.html" rel="noopener nofollow ugc">here</a>). I’m confused on how this function still trains the linear regression model since it calls for loss.backward() after loss.no_grad()?<br>
Thank you so much for your help!</p>
<pre><code class="lang-auto">@d2l.add_to_class(d2l.Trainer)  #@save
def fit_epoch(self):
    self.model.train()
    for batch in self.train_dataloader:
        loss = self.model.training_step(self.prepare_batch(batch))
        self.optim.zero_grad()
        with torch.no_grad():
            loss.backward()
            if self.gradient_clip_val &gt; 0:  # To be discussed later
                self.clip_gradients(self.gradient_clip_val, self.model)
            self.optim.step()
        self.train_batch_idx += 1
    if self.val_dataloader is None:
        return
    self.model.eval()
    for batch in self.val_dataloader:
        with torch.no_grad():
            self.model.validation_step(self.prepare_batch(batch))
        self.val_batch_idx += 1
</code></pre>

username: ptrblck
cooked: <p>The code looks indeed a bit confusing:</p>
<pre data-code-wrap="python"><code class="lang-python">        with torch.no_grad():
            loss.backward()
            if self.gradient_clip_val &gt; 0:  # To be discussed later
                self.clip_gradients(self.gradient_clip_val, self.model)
            self.optim.step()
</code></pre>
<p>You don’t need to call <code>loss.backward()</code> and <code>self.optim.step()</code> in a <code>no_grad</code> context.<br>
I don’t know if <code>self.clip_gradients</code> is performing inplace manipulation of parameters (I would not think so) and guess the <code>no_grad</code> context can be removed.</p>

username: hanxix
cooked: <p>Hi, thank you so much for your help. Could you help me understand how the original function can still train a model with torch.no_grad being called? Thank you!!</p>

username: ptrblck
cooked: <p>The forward pass is outside the <code>no_grad</code> context, so the computation graph will be created and all intermediate activations, needed for the gradient computation, will be saved as well.</p>

username: hanxix
cooked: <p>thank you so so much! This is very helpful!</p>
Cuda 12.6 supported ?

username: xemanx
cooked: <p>Hello,<br>
I would like to know if Pytorch supports cuda 12.6 as I am having malfunctions.</p>
<p>Thanks in advance</p>

username: ptrblck
cooked: <p>Yes, it does and you can download the nightly PyTorch binaries with CUDA 12.6.3.</p>
Best way to obtain first occurance in a tensor

username: mrmoody
cooked: <p>Hey everyone,</p>
<p>could someone tell me the best way to get the indices of the first occurrences in a tensor along a specific dimension? By occurrence I mean the indices where the tensor is e.g. larger than a specific value:</p>
<p>For example, consider</p>
<pre><code class="lang-auto">my_tensor = torch.rand(10, 100)
my_value = 0.5

ind = (my_tensor &lt; my_value) # Boolean mask whether the values are &lt; 0.5
</code></pre>
<p>How do I obtain the first occurrence where ind == 1 along the dimension -1 (in a (10, 1) tensor) ?<br>
For example, ind.nonzero() returns all non-zero values and does not keep the dimensions…</p>
<p>Thanks a lot!</p>

username: MariosOreo
cooked: <p>Hi Michael,</p>
<p>In my shallow view, the <code>.nonzero</code> operation cannot returns the <code>keepdim=True</code> version, because it cannot guarantee each dimension has the same number of elements that fit the nonzero condition, so it performs reduction to 1D and return positions.</p>
<p><strong>But</strong> you could get the first occurrence by iteration like this:</p>
<pre><code class="lang-auto">next(x for x in nz_ind if x[0] &gt; 0)
</code></pre>
<p>to get the first nonzero value in dim1.</p>

username: sachinruk
cooked: <p>bump. Is there a better answer for this in 2021?</p>

username: lassemj
cooked: <p>As this is still getting some views, here is another approach.</p>
<pre><code class="lang-auto">my_tensor = torch.rand(10, 100)
my_value = 0.5

# step by step
bool_mask = (my_tensor &lt; my_value) # Boolean mask with your condition (here &lt; 0.5 but can be anything)```
int_mask = bool_mask.int() # argmax is not working on bool
idx = torch.argmax(int_mask, dim=-1) # argmax returns the first maximum value along a dimension.

# one-liner
idx = (my_tensor &lt; my_value).int().argmax(dim=-1)</code></pre>
FSDP memory higher than DDP or single gpu

username: ening
cooked: <p>I find this problem. When I use fsdp and three gpus to fine-tune llama3 vision model, I find every gpu(fsdp) has a higher memory than others(every gpu in ddp and single gpu). I get OOM error and it happens in _flat_param.py when flattening tensors. What’s more, even I can load model when using fsdp, it has higher memory than ddp.<br>
It may be a bug in pytorch?</p>

username: ptrblck
cooked: <p>Did you try <code>FSDP2</code>, as it’s supposed to use less memory? From the <a href="https://github.com/pytorch/pytorch/issues/114299">RFC</a>:</p>
<blockquote>
<p>We have validated the prototype on Llama-like models, achieving on-par throughput while using less memory.</p>
</blockquote>

username: ening
cooked: <p>I only have used FSDP because I refer from llama-recipes to write my own finetuning.py. Thank you for your suggestion, I will try it!</p>

username: XYG_123
cooked: <p>Thanks for reply. I tried FSDP2 and it definitely has better compatibility with other features, like activation checkpointing and torch compile.</p>
<p>I can see my gpu mem saving now.</p>
Can't find method of a model

username: yezi
cooked: <p>model is successfully loaded but have no methods such as forward and others,model is transferred by sevenn_get_model( <a href="https://github.com/MDIL-SNU/SevenNet/blob/a1fb4c44e767055f822e5e2d3817f4214f65c64b/README.md" rel="noopener nofollow ugc">SevenNet/README.md at a1fb4c44e767055f822e5e2d3817f4214f65c64b · MDIL-SNU/SevenNet</a>)</p>
<p>code for load model:<br>
std::unordered_map&lt;std::string, std::string&gt; meta_dict = {<br>
{“chemical_symbols_to_index”, “”},<br>
{“cutoff”, “”},<br>
{“num_species”, “”},<br>
{“model_type”, “”},<br>
{“version”, “”},<br>
{“dtype”, “”},<br>
{“time”, “”}};</p>
<pre><code>static bool model_loaded = false;
if (!model_loaded){
    model = torch::jit::load(model_path, device, meta_dict);
    model_loaded = true;
}
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/f/efd498d17c4eb431ff533fb42ad5437570f10b50.png" data-download-href="https://discuss.pytorch.org/uploads/default/efd498d17c4eb431ff533fb42ad5437570f10b50" title="97b8ab0b4f0f1d033ca9c0ca9937e444"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/f/efd498d17c4eb431ff533fb42ad5437570f10b50.png" alt="97b8ab0b4f0f1d033ca9c0ca9937e444" data-base62-sha1="ydDBldE2aiGAvtqvJaA4LueMYRG" width="690" height="216" data-dominant-color="202621"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">97b8ab0b4f0f1d033ca9c0ca9937e444</span><span class="informations">1297×407 24.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
My Accuracy remains constant

username: arsh_sharma
cooked: <p>Hi, I am trying to do sentence classification 0/1 with LSTM. However, my model is stuck and giving same accuracy even when loss is decreasing. Please find out the error in my code. Attaching my colab file <a href="https://colab.research.google.com/drive/1G-7Upf-JfNYjdboCsmaGDHimw2hsWCwb#scrollTo=31rvLry2SU-K" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1G-7Upf-JfNYjdboCsmaGDHimw2hsWCwb#scrollTo=31rvLry2SU-K</a></p>

username: shubham-attri
cooked: <p>Your Loss value’s are not changing means your backward pass might not be working fine and the weights are not getting updated properly.</p>
<p>Try looking at parameter values for each epoch.</p>
Does warming up the torch.compile necessarily trigger all cases?

username: GodHforever
cooked: <p>my test code is as follows:</p>
<pre><code class="lang-auto">@torch.compile(backend=my_compiler)
def my_function(inputs):
    x = inputs["x"]
    y = inputs["y"]
    x = torch.add(x, 0.5)
    if x.mean() &gt; 0.5:
        x = x / 1.1
    return x * y
</code></pre>
<p>I debug into the compile process and find that tensor <code>x</code> always save the true value instead of fake tensor, thus different inputs can trigger all code branches.<br>
In theory, <code>x</code> should be a fake tensor, so is it possible that no mattter how many times you warm up models, it will never trigger some corner branches?</p>

username: bdhirsh
cooked: <p>The direct answer to your question is no: torch.compile is a just-in-time, tracing-based compiler, so in order to fully “warm up” compile (and avoid new compilations / reach a steady state), you need to sure you warm it up with a full set of model inputs that go down all potential branches of your model.</p>
<p>On your specific example: when you branch on “x.mean() &gt; 0.5”, you are attempting to branch on a data-dependent value. Compile will capture a graph up until the branch, compile and execute it, and then use the computed value to “continue” and capture a new graph on the side of the control flow that the value sends it.</p>
<p>If you want to see what the actual graphs look like, you can run with “TORCH_LOGS=graph_code” to see the two distinct graphs that compile produces</p>

username: GodHforever
cooked: <p>Thanks for your reply. dynamo says it will use faketensors which only include meta data, by in this case real data is captured.</p>
<p>Does this mean that if there is a computational process that is data-dependent, then the values of the real inputs are used, otherwise the faketensor is used?</p>
FSDP issue with invertible networks

username: gseastream
cooked: <p>I am new to FSDP and trying to implement it for my model training (replacing my DDP usage). My model is a normalizing flow (implemented with flowtorch), which is an invertible network. The train step looks like this:</p>
<pre data-code-wrap="python"><code class="lang-python">def train_step(model: FSDP, batch_size: int):
    ...
    loss = model(batch_size)
    ...

class Model(nn.Module):
    ...

    def forward(self, batch_size):
        return calc_loss(self, batch_size)

def calc_loss(model, batch_size):
    samples = model.sample([batch_size])
    log_prob = model.log_prob(samples)
    return -log_prob
</code></pre>
<p>The normalizing flow is a composition of (trainable) bijectors. The sample method runs the flow forwards to produce random samples; it calls bijector.forward on each bijector. The log_prob method runs the flow in reverse to compute the probability densities of those samples; it calls bijector.inverse on each one.</p>
<p>The forward (sample) is fine. But when it does bijector.inverse (for the log_prob), I get errors suggesting that the shards were not gathered (“mat2 must be a matrix, got 1-D tensor”). Even when I just run the log_prob method outside the train_step (by hijacking the forward method of the model to call log_prob instead of calc_loss), I still get errors because the shards were not gathered.</p>
<p>Is there a solution to this? I would like to shard my parameters if possible, but I don’t know how to get FSDP to gather the shards again when I start running the units in reverse order to compute the log_prob. Right now I can only get it to run with <code>ShardingStrategy.SHARD_GRAD_OP</code>.</p>
<p>Thanks in advance!</p>

username: gseastream
cooked: <p>Ok… Well my problem kinda fixed itself. I was experimenting with a stripped down version of my model, and I found out that if I set min_num_params to 1e6 instead of 1e7, the problem went away. I also changed up my code so that all my parameters have requires_grad equal to True, and made everything else just a buffer instead of a parameter. This was in an effort to set use_orig_params to False. Well, now everything works… Even if I go back to 1e7, or if I pass use_orig_params equals True.</p>
<p>Apparently this had nothing to do with invertible networks. Maybe it was an issue with parameters that didn’t require grads. Who knows!</p>
Torch version for python-3.10.8

username: Frank_Liu
cooked: <p>I installed torch-2.5.1, torchaudio-2.5.1 and torchvision-0.20.1 in python-3.10.8, CUDA/12.0.0, GCCcore-12.2.0 on Linux. But there was an error when I imported torch in python. Please could you let me know what versions of torch, torchaudio and torchvision are compatible with my system?</p>

username: ptrblck
cooked: <p>What kind of error did you see?</p>

username: Frank_Liu
cooked: <p>Hey, I downgrade pytorch to 2.1.2 and it is working on CUDA/12.1 now. Thanks.</p>
Does pytorch support building from source on arrch64

username: 2209262843
cooked: <p>Does pytorch support building from source on arrch64?<br>
when I build pytorch2.1.2 cu118 with python3.9, get error about"avx".but I saw others build gpu pytorch v2.0.1 version succeed <a href="https://blog.csdn.net/hknaruto/article/details/132692665" rel="noopener nofollow ugc">https://blog.csdn.net/hknaruto/article/details/132692665</a></p>

username: ptrblck
cooked: <p>Yes, PyTorch supports ARM and you can download pre-built binaries for Jetson devices, ARM servers with NVIDIA GPUs, newer Macs, and CPU-only ARM builds.</p>

username: 2209262843
cooked: <p>thanks，I did it，but I have another question:I am compiling the torch gpu version on the ARM architecture, but I find that the computing power of gpu and glibc cannot be reused. For example, the whl package of torch compiled on the computing power sm70 cannot be used on other computing gpu. The whl package compiled on glibc2.3.30 cannot be used on operating systems earlier than this glibc version. Is there a way to compile a relatively general torch?</p>

username: ptrblck
cooked: <p>Yes, add the desired GPU architecture via <code>TORCH_CUDA_ARCH_LIST</code> and use e.g. a <code>manylinux</code> environment with a low glibc to build the wheels.</p>