akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
DDP - sync gradients during optim step instead of backward

username: benoriol
cooked: <p>If I understood correctly, DDP wrapper on my model synchronizes the gradients at every backward pass. This seems to cause a massive bottleneck performance due to communication between gpus at every minibatch.<br>
Because I am accumulating gradients over multiple mini-batches, I would like the gradient synchronisation to happen at every optimizer step instead of minibatch backward pass, which would massively speed up my training and be theoretically equivalent.</p>
<p>Questions are:</p>
<ol>
<li>is my understanding of DDP correct?</li>
<li>If so, do you have any hint on how to sync gradients during backward pass only?<br>
Thanks in advance.</li>
</ol>
<p>EDIT: Is this claude solution the best way to go about it?</p>
<pre data-code-wrap="python"><code class="lang-python"># Setup
model = YourModel().to(device)
model = DDP(model, device_ids=[local_rank], find_unused_parameters=False)
optimizer = torch.optim.Adam(model.parameters())

# Hyperparameters
accumulation_steps = 4  # Number of batches to accumulate before synchronizing
optimizer.zero_grad()

for epoch in range(num_epochs):
    for i, (inputs, targets) in enumerate(dataloader):
        # Forward pass
        outputs = model(inputs)
        loss = loss_fn(outputs, targets)
        
        # Scale loss by accumulation steps
        loss = loss / accumulation_steps
        
        # Backward pass - DDP will NOT sync at this point if we disable it
        with model.no_sync():  # This is the key part - disables synchronization
            loss.backward()
        
        # Only perform synchronization and optimizer step after accumulation_steps
        if (i + 1) % accumulation_steps == 0:
            # Re-enable synchronization for the final backward pass
            # Or perform synchronization manually
            for param in model.parameters():
                if param.requires_grad and param.grad is not None:
                    dist.all_reduce(param.grad.data, op=dist.ReduceOp.SUM)
                    param.grad.data = param.grad.data / world_size
            
            # Now update parameters
            optimizer.step()
            optimizer.zero_grad()
</code></pre>

username: ptrblck
cooked: <p>You could use the <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel.no_sync"><code>no_sync</code></a> context manager to delay the gradient synchronization and avoid it during gradient accumulation.</p>
How to set OMP_NUM_THREADS for distruted training?

username: Brando_Miranda
cooked: <p>I got a warning but there was no link or suggestions of how to tune this number (or what it means). How do I choose this value?</p>
<p>Error:</p>
<pre><code class="lang-auto">(meta_learning_a100) [miranda9@hal-dgx diversity-for-predictive-success-of-meta-learning]$ python -m torch.distributed.launch --nproc_per_node=2 ~/ultimate-utils/tutorials_for_myself/my_l2l/dist_maml_l2l_from_seba.py

/home/miranda9/miniconda3/envs/meta_learning_a100/lib/python3.9/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated
and will be removed in future. Use torch.distributed.run.
Note that --use_env is set by default in torch.distributed.run.
If your script expects `--local_rank` argument to be set, please
change it to read from `os.environ['LOCAL_RANK']` instead. See 
https://pytorch.org/docs/stable/distributed.html#launch-utility for 
further instructions

  warnings.warn(
WARNING:torch.distributed.run:*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
</code></pre>

username: Dazitu616
cooked: <p>Since it’s an environment variable, I think you can simply set its value by:<br>
<code>OMP_NUM_THREADS=$VALUE python -m torch.distributed.launch --nproc_per_node=2 xxxxx</code><br>
This is similar to other environment variables e.g. <code>CUDA_VISIBLE_DEVICES</code></p>

username: Brando_Miranda
cooked: <p>oh cool. But I was curious about how does one choose the value of <code>OMP_NUM_THREADS=$VALUE</code></p>

username: likaizhou8
cooked: <p>bro，did you have slove this problem?</p>

username: Kenzo
cooked: <p>It seems that an optimal value can be found by OMP_NUM_THREADS = nb_cpu_threads / nproc_per_node. Use htop to know about number of CPU threads</p>

username: handaru
cooked: <h1><a name="p-465171-set-omp-threads-to-number-of-logical-cpus-1" class="anchor" href="#p-465171-set-omp-threads-to-number-of-logical-cpus-1"></a>set omp threads to number of logical cpus</h1>
<p>import os</p>
<p>logical_cpus = os.process_cpu_count()</p>
<p>print(logical_cpus)</p>
<p>os.environ[‘OMP_NUM_THREADS’] = str(logical_cpus)</p>

username: MinSnz
cooked: <p><code>OMP_NUM_THREADS</code> should be set to <code>#PhysicalCores / #Processes</code>.</p>
<ul>
<li><code>os.process_cpu_count</code>, as <a class="mention" href="/u/handaru">@handaru</a> mentioned and was <a href="https://docs.python.org/3/library/os.html#os.process_cpu_count" rel="noopener nofollow ugc">added in Python 3.13</a>, as well as <code>nproc</code> and <code>htop</code>, gives the number of <em>logical</em> CPUs. Please use <code>psutil.cpu_count(logical=False)</code> to get the number of <strong>physical</strong> CPUs.</li>
<li>Don’t forget to divide it by <code>nproc_per_node</code> as <a class="mention" href="/u/kenzo">@Kenzo</a> mentioned, or you’ll have many processes competing for each core.</li>
</ul>
pytorch profiler to analyze memory consumption

username: 111418
cooked: <p>hi.</p>
<p>I’m using a pytorch profiler to analyze memory consumption.</p>
<p>I want to know the units.</p>
<p>Mb, Kb, b</p>
<p>Does this means bytes?? or bits?<br>
Mega bytes? or Mega bits?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/6/866628d14d85c86000bdaf8877a4cdeccfcef546.png" alt="image" data-base62-sha1="jaWR6V1OVZ784yGi0HOiVYyLdI2" width="672" height="186"></p>

username: Usama_Hasan
cooked: <p>Hy there, the answer is quite clear, we always consider bytes as unit of memory consumption of our primitive datatypes.</p>

username: Agustin_Barrachina
cooked: <p>No it’s not, Normally ‘B’ is used for Bytes and ‘b’ is used for bits.<br>
<a href="https://www.google.com/search?q=bits%20vs%20bytes%20notation" rel="noopener nofollow ugc">bits vs bytes notation - Google Search</a></p>
CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacty of 3.69 GiB of which 20.81 MiB is free

username: Osama_Sleem
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad0f0faeea634bbcc0a7fff0982c348a6bc44983.png" data-download-href="https://discuss.pytorch.org/uploads/default/ad0f0faeea634bbcc0a7fff0982c348a6bc44983" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad0f0faeea634bbcc0a7fff0982c348a6bc44983_2_690x328.png" alt="image" data-base62-sha1="oGWS06kwKVd64FMcKk9OVQJHH6H" width="690" height="328" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/d/ad0f0faeea634bbcc0a7fff0982c348a6bc44983_2_690x328.png, https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad0f0faeea634bbcc0a7fff0982c348a6bc44983.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/d/ad0f0faeea634bbcc0a7fff0982c348a6bc44983.png 2x" data-dominant-color="39142D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">940×447 42.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Soumya_Kundu
cooked: <p>We require some more context.</p>
<p>The error specifically indicates that you are putting more memory than your GPU can handle.</p>
OSError when import torch

username: MIA_D
cooked: <p>Hello, to whom it concerns<br>
When trying to use pybind11 to encapsulate a python program, an error occurs when importing torch,no problem when importing other libraries.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/5/754e5bd410b819b654264ea16ed4d2cf27b648fa.png" data-download-href="https://discuss.pytorch.org/uploads/default/754e5bd410b819b654264ea16ed4d2cf27b648fa" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/5/754e5bd410b819b654264ea16ed4d2cf27b648fa.png" alt="image" data-base62-sha1="gJJKiFENhCGsZtCJMsThkCzJ4qm" width="690" height="251" data-dominant-color="1E1E1E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1098×401 18.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
When use offload_to_cpu=True to fsdp load, the usage of video memory still increased

username: nobody
cooked: <pre data-code-wrap="python"><code class="lang-python">log_gpu_memory_usage('fsdp checkpoint load 1.2', logger=None)

        state_dict_cfg = ShardedStateDictConfig(offload_to_cpu=True)
        optim_cfg = ShardedOptimStateDictConfig(offload_to_cpu=True)
        with FSDP.state_dict_type(self.model, StateDictType.SHARDED_STATE_DICT, state_dict_cfg, optim_cfg):
            self.model.load_state_dict(model_state_dict)
            if self.optimizer is not None:
                self.optimizer.load_state_dict(optimizer_state_dict)
        log_gpu_memory_usage('fsdp checkpoint load 1.3', logger=None)
</code></pre>
<p>I obtained the following printing results.</p>
<pre><code class="lang-auto">fsdp checkpoint load 1.2, memory allocated (GB): 52.07300519943237, memory reserved (GB): 61.4921875
fsdp checkpoint load 1.3, memory allocated (GB): 65.82867097854614, memory reserved (GB): 67.294921875
</code></pre>
<p>I think this doesn’t meet the expectations. Why does the loaded parameters increase the usage of GPU video memory? Is this a bug? Or is there something that I’m not aware of?</p>

username: nobody
cooked: <p><span class="mention">@ptrblck_de</span> Is it possible to provide some assistance, possible causes or solutions?</p>
Some questions about running pytorch on an intel "arc graphics xpu"

username: KFrank
cooked: <p>Hi All!</p>
<p>In short, would I expect to be able to install and run pytorch for the intel xpu relatively<br>
easily and would I expect such a version of pytorch not to conflict with my “regular”<br>
cuda version?</p>
<p>I have a laptop that has an “ultra 9 185h” processor that supports “arc graphics,” as<br>
well as an rtx 3000 laptop gpu.</p>
<p>This documentation, <a href="https://pytorch.org/docs/main/notes/get_start_xpu.html" rel="noopener nofollow ugc">“Getting Started on Intel GPU,”</a> suggests that I should be able<br>
to run pytorch on the intel “xpu.”</p>
<p>Am I right that there is not a pre-built distribution that supports cuda and xpu tensors<br>
simultaneously?  (As an aside, would it be currently practical build such a version, or<br>
does the current structure of pytorch not support using both cuda and xpu tensors in<br>
the same pytorch instance?)</p>
<p>If I were to install a separate version of pytorch for the intel xpu, I would presumably<br>
do so in a separate conda environment.  (I routinely run different versions of pytorch<br>
in different conda environments without issue.)</p>
<p>If I were to install cuda and xpu versions of pytorch in separate conda environments,<br>
would it be likely that they would not conflict with one another?</p>
<p>Thanks for any pointers about this.</p>
<p>K. Frank</p>
Fitting parameters of custom model with pytorch

username: fracton
cooked: <p>Hi,</p>
<p>I would like to fit parameters of a custom model using the optimizers in pytorch. For example, say I have a model function yi=F(Xi, beta) where Xi is my input array (37x4) and beta is the parameter array. How can use the stochastic gradient descent algorithm to perform regression analysis using a dataset.</p>

username: KFrank
cooked: <p>Hi Fracton!</p>
<aside class="quote no-group" data-username="fracton" data-post="1" data-topic="217919">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/dbc845/48.png" class="avatar"> fracton:</div>
<blockquote>
<p>How can use the stochastic gradient descent algorithm to perform regression analysis using a dataset.</p>
</blockquote>
</aside>
<p>I would recommend that you work through some of the <a href="https://pytorch.org/tutorials/index.html" rel="noopener nofollow ugc">pytorch tutorials,</a> perhaps<br>
starting with <a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html" rel="noopener nofollow ugc">this one.</a> Pay attention to its <a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#autograd" rel="noopener nofollow ugc">section on autograd</a> and then go through<br>
<a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-custom-nn-modules" rel="noopener nofollow ugc">this section</a> to see how to use stochastic gradient descent (<code>optim.SGD</code>) to fit a<br>
parametrized function to some data by optimizing its parameters.</p>
<p>Good luck!</p>
<p>K. Frank</p>
PyTorch for RTX 5090? When will it be out? Thank you

username: Raf_Duran
cooked: <p>PyTorch for RTX 5090?? When will it be out? Thank you</p>

username: ptrblck
cooked: <p>Please use the search function instead of cross-posting the same questions: <a href="https://discuss.pytorch.org/t/5070ti-ubuntu-20-04-6-cuda/217901/2" class="inline-onebox">5070Ti+Ubuntu 20.04.6+cuda? - #2 by ptrblck</a></p>
This is about optimizing cuDNN to avoid rebuilding the graph when batch, input dimensions change.

username: yhyang201
cooked: <p>According to the code in PyTorch at <a href="https://github.com/pytorch/pytorch/blob/a9c55277d7589f5fb07e9ffb46bf6bb198e0af7f/aten/src/ATen/native/cudnn/MHA.cpp#L406" rel="noopener nofollow ugc">this link</a>, it shows that when running the SDPA with cuDNN multiple times, if the batch size (B) or sequence length (S) of the query, key, or value changes, the cudnn graph needs to be rebuilt each time. This is quite time-consuming. Could you kindly advise if there is any way to avoid rebuilding the cudnn graph multiple times?</p>
Storing intermediates outputs in python list + clone() and memory usage problem

username: Nyx
cooked: <p>Hi,<br>
I have a CNN block composed of a graph of conv layers, when doing a forward pass of this block, the intermediates layers outputs are stored in a list of lists of tensors (representing the graph matrix), however when testing I get an error that I believe is related to in-place operations and gradient calculation which I resolved by storing the clone() of the intermediates outputs but the problem is that now each block uses <strong>2 times more memory</strong> when the forward method is called.<br>
Is there another way to solve this? any suggestions on how to modify the code would be welcome</p>
<pre><code class="lang-auto">class Block(nn.Module):
  def __init__(self):
          super(Block, self).__init__()
          ...
  def forward(self, X):
    intermediates = [[] for _ in range(self.block["num_nodes"])]
    
    X_clone = X.clone()
    intermediates[0].append(X_clone)
               
    for i in range(self.block["num_nodes"]):
                    
                    Z = intermediates[i]
                    for j in range(self.block["num_nodes"]):
                        if self.layers[i][j] is not None:
                            branches = self.layers[i][j]
                            for branch in branches:
                                    Z_clone = branch(Z).clone()
                                    intermediates[j].append(Z_clone)
    
                output1 = torch.cat(intermediates[self.block["num_nodes"] - 1], dim=1)
                output2 = self.pool_op(output1)
    return output2
</code></pre>
Pruning by manipulating weight_mask

username: ndronen
cooked: <p>Should one be able to enable (disable) pruning of specific pruned neurons by assigning 0 (1) to the corresponding elements of weight_mask? I’ve tried this script with torch 2.6 and 2.5.1.</p>
<pre><code class="lang-auto">import torch
import torch.nn.utils.prune as prune

weight = torch.arange(1, 5).repeat(1, 4).view(4, 4).float()

linear1 = torch.nn.Linear(4, 4, bias=False)
linear2 = torch.nn.Linear(4, 4, bias=False)

linear1.weight.data = weight.clone()
linear2.weight.data = weight.clone()

prune.ln_structured(linear1, name="weight", n=2, amount=0.5, dim=0)
assert linear1.weight_mask.count_nonzero() == 8
assert linear1.weight.count_nonzero() == 8

prune.identity(linear2, "weight")
linear2.weight_mask[:2].fill_(0)
assert linear2.weight_mask.count_nonzero() == 8 # pass
assert linear2.weight.count_nonzero() == 8      # fail
</code></pre>

username: ndronen
cooked: <aside class="quote no-group" data-username="ndronen" data-post="1" data-topic="216953">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ndronen/48/1575_2.png" class="avatar"> ndronen:</div>
<blockquote>
<pre><code class="lang-auto">import torch
import torch.nn.utils.prune as prune

weight = torch.arange(1, 5).repeat(1, 4).view(4, 4).float()

linear1 = torch.nn.Linear(4, 4, bias=False)
linear2 = torch.nn.Linear(4, 4, bias=False)

linear1.weight.data = weight.clone()
linear2.weight.data = weight.clone()

prune.ln_structured(linear1, name="weight", n=2, amount=0.5, dim=0)
assert linear1.weight_mask.count_nonzero() == 8
assert linear1.weight.count_nonzero() == 8

prune.identity(linear2, "weight")
linear2.weight_mask[:2].fill_(0)
assert linear2.weight_mask.count_nonzero() == 8 # pass
assert linear2.weight.count_nonzero() == 8      # fail
</code></pre>
</blockquote>
</aside>
<p>I figured this out. Updating <code>weight_mask</code> manually is only reflected in the <code>weight</code> tensor after calling the module, which will invoke the forward pre-hook registered by the pruning framework.</p>
<pre><code class="lang-auto">import torch
import torch.nn.utils.prune as prune

weight = torch.arange(1, 5).repeat(1, 4).view(4, 4).float()
linear2 = torch.nn.Linear(4, 4, bias=False)
linear2.weight.data = weight.clone()
prune.identity(linear2, "weight")
linear2.weight_mask[:2].fill_(0)
linear2(torch.randn(1, 4))
assert linear2.weight_mask.count_nonzero() == 8  # pass
assert linear2.weight.count_nonzero() == 8       # pass
</code></pre>
Simple CNN model takes too much memory when forward() is called

username: Nyx
cooked: <p>Hello,<br>
In the following code, I define a simple CNN model composed of:<br>
CNN(256 out-channels, 5x5 kernel) → ReLU → Flatten()</p>
<p>when I run the model with a sample of input data, torch.cuda.memory_allocated() shows 1.8Gb allocated, is that normal? if not why is it so high? what am I doing wrong?</p>
<p>the code:<br>
(PS cfg is a config file)</p>
<pre><code class="lang-auto">import cfg
import torch
from torch import nn
import gc

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.layer1 = nn.LazyConv2d(out_channels=256, kernel_size=5)
        self.layer2 = nn.Flatten()
        self.model = nn.Sequential(self.layer1, nn.ReLU() ,self.layer2)

    def forward(self, X):

        print("Before model call")
        print("torch.cuda.memory_allocated: %fGB"%(torch.cuda.memory_allocated(0)/1024/1024/1024))

        out = self.model(X)

        print("\n\nAfter model call")
        print("torch.cuda.memory_allocated: %fGB"%(torch.cuda.memory_allocated(0)/1024/1024/1024))
        

        return out


model = Model()


model.to(cfg.DEVICE)


gc.collect()
torch.cuda.empty_cache()

X = torch.randn(size=(8, 1, 500, 500)).to(cfg.DEVICE)


output = model(X)
</code></pre>
<p>the output:</p>
<pre><code class="lang-auto">Before model call
torch.cuda.memory_allocated: 0.007451GB


After model call
torch.cuda.memory_allocated: 1.884429GB
</code></pre>

username: ptrblck
cooked: <p>The memory usage is expected as already the output tensor consumes almost all of the used memory:</p>
<pre data-code-wrap="python"><code class="lang-python">print(output.nelement() * output.element_size() / 1024**3)
# 1.876953125
</code></pre>
Help Needed: High Inference Time & CPU Usage in VGG19 QAT model vs. Baseline

username: Auniik
cooked: <p>Hey everyone,</p>
<p>I’m working on improving a model based on VGG19 Baseline Model with CIFAR-10 dataset and noticed that my modified version has significantly higher inference time and CPU usage. I was expecting some overhead due to the changes, but the difference is much larger than anticipated.</p>
<p>I’ve been troubleshooting for a while but haven’t been able to pinpoint the exact issue.</p>
<p>If anyone with experience in optimizing inference time and CPU efficiency could take a look, I’d really appreciate it!</p>
<p>My notebook link with the code and profiling results:</p>
<p><a href="https://colab.research.google.com/drive/1g-xgdZU3ahBNqi-t1le5piTgUgypFYTI" rel="noopener nofollow ugc">Colab notebook link</a></p>
Pytorch Dataset why so slow

username: Yi_Cheng1
cooked: <p>I extended DataSet, class CyDataSet(Dataset).<br>
basically implements reading data from files.<br>
in init i loaded all files to memory,<br>
in getitem, i translated the index to my-cached df, then directly index into the df and return.<br>
def <strong>getitem</strong>(self, index):<br>
ssidx = bisect.bisect_right(self.cumulative_sums, index)<br>
ss = self.sslist[ssidx]<br>
space_start_index = self.cumulative_sums[ssidx - 1] if ssidx &gt; 0 else 0<br>
relative_position = index - space_start_index<br>
features_tensors, target_tensors = self.pdfCache[file_key]<br>
X = features_tensors[relative_position]<br>
Y = target_tensors[relative_position]<br>
symbol_id = self.symbolTensorMap[ss.symbol]<br>
return X, symbol_id, Y<br>
Just this simple, for batch size=2048, for 1 batch, getitem take around 1.2s, traning and backpro take 1s on CPU. why getitem so slow? any other framework code involved, why no batch_getitem?</p>

username: ptrblck
cooked: <p>You could profile your code to narrow down where the bottleneck is as it’s unclear to me if the <code>getitem</code> call itself is slow or any of the calls you are using inside.</p>
Torch::fft::rfft and fftw library disagree

username: Himalayjor
cooked: <p>Torch implementation</p>
<pre><code class="lang-auto">torch::fft::rfft(tensor).abs().pow(2.0);
</code></pre>
<p>Rough FFTW implementaton</p>
<pre><code class="lang-auto">fftwf_plan plan = fftwf_plan_dft_r2c_2d(N0, N1, input, output, FFTW_ESTIMATE);

for (int i = 0; i &lt; N0; ++i) {
        for (int j = 0; j &lt; N1 / 2 + 1; ++j) {
            float real_part = output[i * (N1 / 2 + 1) + j][0];
            float imag_part = output[i * (N1 / 2 + 1) + j][1];
            power.index({ i,j }) = real_part * real_part + imag_part * imag_part;
        }
    }

return power
</code></pre>
How to ensure that auto-generated Triton Kernel is executed

username: deepak.vij
cooked: <p>Hi all, is there a way to ensure that the auto-generated Triton kernels are executed at runtime. I have set the following environment variables. Which log file to look at in order to ensure that if the auto-generated Triton code is executed versus native PyTorch kernel code. Thanks.</p>
<p>export TORCH_LOGS=“output_code”<br>
export TORCH_COMPILE_DEBUG=1<br>
export TORCH_COMPILE_VERBOSE=1<br>
export INDUCTOR_LOG_LEVEL=“INFO”</p>
NVIDIA RTX 5070 with CUDA 12.8 (sm_120) Window 11 Error

username: adventer705
cooked: <p>I am using an **NVIDIA RTX 5070 with CUDA 12.8 (sm_120)  Window 11 ** and trying to use <strong>GPT-SoVITS</strong>, but I encountered a compatibility error that prevented it from running.</p>
<p>NVIDIA GeForce RTX 5090 with CUDA capability sm_120 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_61 sm_70 sm_75 sm_80 sm_86 sm_90.<br>
If you want to use the NVIDIA GeForce RTX 5090 GPU with PyTorch, please check the instructions at <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>
<p>Yesterday, I updated to the preview (Nightly) version, which was recently released(torchvision, torchaudio). However, during the update process, the following error message appeared.</p>
<p>Could you help me resolve this issue?</p>
<hr>
<p>ERROR: pip’s dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.<br>
fairseq 0.12.2 requires hydra-core&lt;1.1,&gt;=1.0.7, but you have hydra-core 1.3.2 which is incompatible.<br>
fairseq 0.12.2 requires omegaconf&lt;2.1, but you have omegaconf 2.3.0 which is incompatible.<br>
torch-directml 0.2.0.dev230426 requires torch==2.0.0, but you have torch 2.7.0.dev20250311+cu128 which is incompatible.<br>
torch-directml 0.2.0.dev230426 requires torchvision==0.15.1, but you have torchvision 0.22.0.dev20250312+cu128 which is incompatible</p>

username: ptrblck
cooked: <p>Remove <code>torch-directml</code> as it conflicts with the nightly PyTorch binary as shown in the error message.</p>
FSDP OOM when forwarding 7B model on 16k context length text

username: zh950713
cooked: <p>I cannot prevent the model from oom using FSDP. I have the knowledge that the 7B model should not OOM with 16k and will oom on 32k. How can I prevent the OOM here?</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.distributed as dist
from torch.distributed.fsdp import FullyShardedDataParallel as FSDP, MixedPrecision, ShardingStrategy, CPUOffload
from transformers.models.qwen2.modeling_qwen2 import Qwen2DecoderLayer, Qwen2Attention, Qwen2MLP, Qwen2ForCausalLM
from transformers import AutoModelForCausalLM, AutoModelForSequenceClassification
from functools import partial
from torch.distributed.fsdp.wrap import transformer_auto_wrap_policy

def get_fsdp_wrap_policy(module):
    if hasattr(module, "get_decoder"):
        transformer_layer_cls = module.get_decoder().layers[0].__class__
        wrap_policy = partial(
            transformer_auto_wrap_policy,
            transformer_layer_cls={transformer_layer_cls},
        )
        return wrap_policy

    if hasattr(module, "layers"):
        transformer_layer_cls = module.layers[0].__class__
        wrap_policy = partial(
            transformer_auto_wrap_policy,
            transformer_layer_cls={transformer_layer_cls},
        )
        return wrap_policy

    raise NotImplementedError(
        "This module is not a decoder-only transformer. " "Please implement the `fsdp_wrap_policy` method."
    )


if __name__ == "__main__":
    length = 16384
    dist.init_process_group(backend="nccl")
    torch.cuda.set_device(dist.get_rank())
    
    input_ids = torch.randint(0, 100, (1, length), device=torch.cuda.current_device())
    position_ids = list(range(length))
    position_ids = torch.tensor(position_ids, device=torch.cuda.current_device()).unsqueeze(0)
    module = AutoModelForCausalLM.from_pretrained(
        "Qwen2/Qwen2.5-7B-Instruct",
        attn_implementation="flash_attention_2",
        device_map="cuda",
        torch_dtype=torch.float32,
    )
    module.gradient_checkpointing_enable()
    model = FSDP(
        module,
        process_group=dist.group.WORLD,
        sharding_strategy=ShardingStrategy.FULL_SHARD,
        auto_wrap_policy=get_fsdp_wrap_policy(module),
        mixed_precision=MixedPrecision(param_dtype=torch.bfloat16),
        device_id="cuda",
        forward_prefetch=True,
        backward_prefetch="BACKWARD_PRE",
        use_orig_params=False,
    )

    # The following will cause OOM
    logits = model(input_ids, position_ids=position_ids).logits
    print(logits.shape, logits.dtype)

    dist.destroy_process_group()
</code></pre>
DDU for segmentation

username: Mohamed_Farag
cooked: <p>Hi everyone!</p>
<p>Has anyone tried to implement the DDU approach for uncertainty quantification for segmentation?</p>
<p>I have checked multiple repos, but still not working for me, specifically fitting the GMM</p>
<p>If someone knows how to get over this, please tell me.</p>
<p>Thanks!</p>
Full finetune, LoRA and feature extraction take the same amount of memory and time to train

username: Vefery
cooked: <p>I need to compare full finetune and LoRA on T5 model on summarization task. The problem is that on my GPU finetune takes ~4.5gb of VRAM and 1 hour to train. And LoRA takes the same amount of memory and time with the same parameters. Sometimes it takes even 100-200mb more according to pytorch. Am I missing something? Do I need to do it different way?<br>
Here’s the snippet of my code. I used to have my own training loop but then opted for Trainer class, though it made no difference.<br>
Finetune setup:</p>
<pre><code class="lang-auto">tokenizer = AutoTokenizer.from_pretrained('t5-base')
model = AutoModelForSeq2SeqLM.from_pretrained('t5-base').to(device)
train_args = TrainingArguments(
    gradient_accumulation_steps=2, 
    gradient_checkpointing=True,
    learning_rate=lr,
    num_train_epochs=epochs,
    warmup_steps=warmup,
    optim='adafactor',
    )
trainer = Trainer(model, train_args, train_dataset=train_dataset, eval_dataset=eval_dataset)
trainer.train()
</code></pre>
<p>LoRA setup:</p>
<pre><code class="lang-auto">config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q", "v"],
    lora_dropout=0.1,
    bias="none",
)
tokenizer = AutoTokenizer.from_pretrained('t5-base')
model = AutoModelForSeq2SeqLM.from_pretrained('t5-base').to(device)
model = get_peft_model(model, config)
train_args = TrainingArguments(
    gradient_accumulation_steps=2, 
    gradient_checkpointing=True,
    learning_rate=lr,
    num_train_epochs=epochs,
    warmup_steps=warmup,
    optim='adafactor',
    )
trainer = Trainer(model, train_args, train_dataset=train_dataset, eval_dataset=eval_dataset)
trainer.train()
</code></pre>
<p>Out of curiosity I tried to freeze all weights except in lm_head (~11%) with <code>require_grad = False</code> and no difference with finetune.</p>
Custom Dataset __getitem__ is receiving a list from DataLoader

username: Gabrichilleron
cooked: <p>Im new using pytorch. I have a custom dataset which is reading images from disk, it looks like this:</p>
<pre><code class="lang-auto">class SaffronDataset(Dataset):
    def __init__(self, base_dir, transform=None, target_transform=None):
        self.images_dir = os.path.join(base_dir, 'raw')
        self.labels_dir = os.path.join(base_dir, 'segmentation')
        self.transform = transform
        self.target_transform = target_transform
        self.images = sorted([os.path.join(self.images_dir,x) for x in os.listdir(self.images_dir)])
        self.labels = sorted([os.path.join(self.labels_dir,x) for x in os.listdir(self.labels_dir)])
        

    def __len__(self):
        return len(self.images)

    def __getitem__(self, idx):
        images = Image.open(self.images[idx])
        labels = Image.open(self.labels[idx])
        if self.transform:
            images = self.transform(images)
        if self.target_transform:
            labels = self.transform(labels)
        return images, labels
</code></pre>
<p>I have a basic DataLoader too:</p>
<pre><code class="lang-auto">data_loader = DataLoader(data, batch_size=2, shuffle=True)
</code></pre>
<p>When i try to iterate the dataloader, im getting the error <strong>TypeError: list indices must be integers or slices, not list</strong>. The problem is that <strong>getitem()</strong> function is receiving a list  instead of a single index and i dont understand why, anyone can help me with that?</p>

username: lostalot
cooked: <p>Might you consider printing out idx so you can see what it contains?</p>

username: Gabrichilleron
cooked: <p>Idx contains a list with a number of index equals to the bacth_size. For example, in my case with batch_size = 2, idx can take [2,3] value.<br>
The problem is that if I take it into account and deal with it in the <strong>getitem</strong>() function, returning a list of images. Then I get an error because a single tuple (image, label) is expected as output.</p>

username: UMAR_MASUD
cooked: <p>Hi, that’s an unusual case. The <code>idx</code> value in <code>__getitem__(self, idx)</code> function should be a single integer value in the range <code>[0,len(dataset)-1]</code> meant for a single sample of the dataset. Getting a list means something is overriding the default behaviour maybe a custom collate function. Do you mind sharing some more code for context and any errors, or print statements?</p>
Torch::jit::load error file_name!=nullptr

username: Sanjib
cooked: <p>Unable to load a model. Model is there, still cannot load the model. Below is my code. Any help appreciated.</p>
<p>Libtorch 2.6 +cu124<br>
Visual Studio 2022</p>
<pre><code class="lang-auto">int LoadModel(const std::filesystem::path&amp; modelPath) {
    try {
        std::cout &lt;&lt; " Loading model from path: " &lt;&lt; modelPath &lt;&lt; std::endl;

        if (!std::filesystem::exists(modelPath)) {
            std::cerr &lt;&lt; "Error: Model file not found at " &lt;&lt; modelPath &lt;&lt; std::endl;
            return -1;
        }
        if (modelPath.empty()) {
            std::cerr &lt;&lt; "Error: Model path is empty!" &lt;&lt; std::endl;
            return -1;
        }
        if (!std::filesystem::exists(modelPath)) {
            std::cerr &lt;&lt; "Error: Model file does not exist at: " &lt;&lt; modelPath &lt;&lt; std::endl;
            return -1;
        }

        std::cout &lt;&lt; "absolute model path: " &lt;&lt; std::filesystem::absolute(modelPath) &lt;&lt; std::endl;

        // Try loading the model
        network = torch::jit::load(modelPath.string());
        network.eval();  // Set to evaluation mode
        std::cout &lt;&lt; "Model loaded successfully!" &lt;&lt; std::endl;
        return 0;
    }
    catch (const c10::Error&amp; e) {
        std::cerr &lt;&lt; " Error loading model: " &lt;&lt; e.what() &lt;&lt; std::endl;
        std::cerr &lt;&lt; "File path: " &lt;&lt; modelPath &lt;&lt; std::endl;
        return -1;
    }
}

int main() {
    std::filesystem::path modelPath = "nws_736_1280_b1.torchscript";  // Path to your model file
    int status = LoadModel(modelPath);

    if (status == 0) {
        std::cout &lt;&lt; "Model is ready for inference!" &lt;&lt; std::endl;
    }
    else {
        std::cerr &lt;&lt; "Failed to load model!" &lt;&lt; std::endl;
    }

    return 0;
}
</code></pre>
Announcement: Share Your Feedback on PyTorch Docs and Tutorial

username: sekyondaMeta
cooked: <p>We’re excited to announce that we’re launching a survey to better understand how you use the PyTorch documentation and tutorials. Your input will help us improve the quality, relevance, and overall user experience of our resources.</p>
<p><strong>Why Participate?</strong></p>
<p>By taking this survey, you’ll be contributing to the development of more effective and user-friendly documentation and tutorials. Your feedback will enable us to:</p>
<ul>
<li>
<p>Identify areas for improvement</p>
</li>
<li>
<p>Develop more relevant and useful content</p>
</li>
<li>
<p>Enhance the overall learning experience for PyTorch users</p>
</li>
</ul>
<p><strong>How to Participate</strong></p>
<p>The survey should only take about 10-15 minutes to complete. You can access it by clicking on the following link:</p>
<p><a href="https://forms.gle/AiJ5ybMnPedEhz1k9" rel="noopener nofollow ugc">https://forms.gle/YcHcF3yBhCmy3NPF9</a></p>
<p><strong>Confidentiality and Data Protection</strong></p>
<p>Your responses will be kept confidential and anonymous. We’ll only use the data collected to improve the PyTorch documentation and tutorials.</p>
<p>We appreciate your time and input in helping us create better resources for the PyTorch community.</p>
<p>Happy surveying!</p>
<p>Best regards,<br>
The PyTorch Docs Team</p>
Torch::full two overloads

username: Dirk10001
cooked: <p>Hello all,</p>
<p>in datasets.cpp in pytorch_cpp-master/utils is the line (473)</p>
<pre><code class="lang-auto">torch::Tensor class_id = torch::full({}, (long int)this-&gt;class_ids.at(idx), torch::TensorOptions().dtype(torch::kLong));
</code></pre>
<p>I am not able to compile this with VS2019.</p>
<p>Neither</p>
<pre><code class="lang-auto">at::Tensor torch::full(at::IntArrayRef,const c10::Scalar &amp;,c10::TensorOptions)
</code></pre>
<p>nor</p>
<pre><code class="lang-auto">at::Tensor torch::full(at::IntArrayRef,const c10::Scalar &amp;,c10::optional&lt;at::DimnameList&gt;,c10::TensorOptions)
</code></pre>
<p>is hit. What is the correct cast at which argument?</p>
<p>Many thanks for your help.</p>

username: Dirk10001
cooked: <pre><code class="lang-auto">auto options = torch::TensorOptions().dtype(torch::kLong);
int val = (long int)this-&gt;class_ids.at(idx);

torch::Tensor class_id = torch::full({}, val, options);
</code></pre>
<p>Does the job.</p>
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 88 but got size 85 for tensor number 1 in the list

username: Dmitr
cooked: <p>I have problem with AI models.<br>
My code:</p>
<pre><code class="lang-auto">import os
from diffusers.callbacks import SDCFGCutoffCallback
from dotenv import load_dotenv
import torch
import numpy as np
from diffusers import AutoPipelineForInpainting, EulerDiscreteScheduler, ControlNetModel, \
    AutoencoderKL, DDIMScheduler
from PIL import Image, ImageOps
import rembg

load_dotenv()
SAVE_PATH = os.getenv('SAVE_PATH')
KANDINSKY_MODEL_PATH = os.getenv('KANDINSKY_MODEL_PATH')
REV_ANIMATED_MODEL_PATH = os.getenv('REV_ANIMATED_MODEL_PATH')
CONTROLNET_MODEL_PATH = os.getenv('CONTROLNET_MODEL_PATH')
MMIX_MODEL_PATH = os.getenv('MMIX_MODEL_PATH')
SAM_MODEL_PATH = os.getenv('SAM_MODEL_PATH')
VAE_MODEL_PATH = os.getenv('VAE_MODEL_PATH')
CANNY_MODEL_PATH = os.getenv('CANNY_MODEL')

prompt = 'best quality, highres, high definition masterpiece, photorealistic, foggy winter forest'
negative_prompt = 'nsfw, people, worst quality, low quality, normal quality, lowres,watermark, monochrome, low resolutio, extra limbs'
num_inference_steps = 40
base_img = 'ai_image2-683x1024.png'
num_of_img_per_prompt = 1

callback = SDCFGCutoffCallback(cutoff_step_ratio=0.4)

def binary_mask(init_image):
    input_array = np.array(init_image)
    mask_array = rembg.remove(input_array, only_mask=True)
    mask_image = Image.fromarray(mask_array)
    mask_image = ImageOps.invert(mask_image)
    return mask_image

def toPill(image):
    minv = np.amin(image)
    maxv = np.amax(image)
    image = image - minv
    image = image / (maxv - minv)
    image = image * 255
    img = Image.fromarray(image.astype(np.uint8))
    return img

if __name__ == '__main__':
    pipeline = AutoPipelineForInpainting.from_pretrained(
        KANDINSKY_MODEL_PATH,
        #controlnet=controlnet,
        use_safetensors=True,
        torch_dtype=torch.float32,
        vae=AutoencoderKL.from_pretrained(
            VAE_MODEL_PATH,
            subfolder=None)
    ).to('cpu')
    init_image = Image.open(base_img)
    init_image = toPill(init_image)
    bin_mask = binary_mask(init_image)

image = pipeline(
        prompt=prompt,
        scheduler=DDIMScheduler.from_config(pipeline.scheduler.config),
        negative_prompt=negative_prompt,
        width=680,
        height=1024,
        num_inference_steps=20,
        image=init_image,
        mask_image=bin_mask,
        control_image=init_image,
        guidance_scale=15,
        strength=1,
        callback_on_step_end=callback
    ).images[0]
    image.save('test1.jpg')
</code></pre>
<p>And i can see this problem in terminal:</p>
<pre><code class="lang-auto">RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 88 but got size 85 for tensor number 1 in the list.
</code></pre>
Stable Diffusion does not work with the 5080 video card

username: Olya
cooked: <p>Hello. I need help.<br>
Given: video card 5080<br>
I run Stability Matrix to use Stable Diffusion and the programme gives an error:</p>
<p>RuntimeError: CUDA error: no kernel image is available for execution on the device<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.<br>
CUDA error: no kernel image is available for execution on the device<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.<br>
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.</p>
<p>Am I right to understand that I can only wait for pytorch update? Or are there other options?<br>
Thank you very much for your reply.</p>

username: ptrblck
cooked: <p>Install the nightly PyTorch binary with CUDA 12.8 and it’ll work.</p>
How to properly implement these 1D convolutions and summations?

username: Sim_On
cooked: <p>How can I properly implement the convolution and summation as shown in the example below?</p>
<p>Lets be given a PyTorch tensor of signals of size <code>(batch_size, num_signals, signal_length)</code>, i.e. each batch contains several signals. For each batch, I want to convolve the i-th signal with the i-th kernel, and sum all of these convolutions. The result should be of shape <code>(batch_size, 1, signal_length)</code></p>
<p>The code snipped below should to the right thing, however I want to avoid unnecessary loops.</p>
<p>Thanks for any help!</p>
<pre data-code-wrap="python"><code class="lang-python">
import torch
import torch.nn as nn
import torch.nn.functional as F

batch_size = 16
num_signals = 5
signal_length = 32
num_kernels = 5
kernel_size = 32

# Random batch of signals and kernels
signals = torch.randn(batch_size, num_signals, signal_length)
kernels = torch.randn(num_kernels, 1, kernel_size)

out = torch.zeros(signals.shape[0], 1, signal_length)
for b in range(signals.shape[0]):
    for k in range(num_signals):
        out[b, 0, :] += F.conv1d(
            signals[b, k].unsqueeze(0).unsqueeze(0),
            kernels[k].unsqueeze(0),
            padding="same"
        ).squeeze()

print("Output shape:", out.shape)

</code></pre>