akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Left padded transformer input with causal mask

StephAO: <p>Hi,<br>
I am training a decoder-style language model using pytorch’s TransformerEncoder. Following huggingface and generally what I’ve found online, I am training using right padding and a causal mask, and during inference I use left padding. My training is working well and if I use teacher forcing during evaluation (i.e. same set up as training), I get what I expect. If I use my inference decoding during evaluation (no teacher forcing) with a batch size of 1, my system works as expected. However, I quickly run into an issue when using left padded sequences during inference decoding. If I include both the causal mask + an attention mask (over the pad tokens), it result in NaN for all sequences that are padded (I think this is because it create tokens that can’t attend to anything with -inf attention which leads to a NaN that then propagates). If I do not include the causal mask during inference, then I get a different output then during (which makes sense, since it is a different set up than training).</p>
<p>So my question is: How can I apply both a causal mask (i.e. kwarg: mask) and an attention mask (i.e. kwarg: src_key_padding_mask) when using left padding?</p>
<p>Thanks in advance for any help.</p>
<p>The snippet of code I am using:</p>
<pre><code class="lang-auto">def forward(self, input_ids: th.Tensor, attention_mask: th.Tensor):
    # Inputs are input_ids and attention_mask, both are th.Tensors
    using_left_pad = th.any(attention_mask[:, 0] == 0)
    if using_left_pad:
        # Adjust position ids to match training which uses right padding
        position_ids = attention_mask.long().cumsum(-1) - 1
        position_ids.masked_fill_(attention_mask == 0, 1)
    # Encodes ids to embeddings, adds positional encodings, and standard normalization
    embeddings, attention_mask = self.preprocessor(input_ids, attention_mask, position_ids=position_ids)
    batch_size, seq_len, d_model = embeddings.shape
        
    # Create causal mask
    causal_mask = self.generate_square_subsequent_mask(seq_len)
    # Attention mask comes from huggingface tokenizer, so requires the 1 - to match the pytorch format
    output = self.decoder(embeddings,  mask=causal_mask, is_causal=True,
                          src_key_padding_mask=(1 - attention_mask).bool())
    # With right padded inputs, there is no issue, with batch size of 1, there is no issue
    # With left padded inputs where batch size &gt; 1, output is all NaNs on padded sequences
</code></pre>

simpleParadox: <p>I’m facing a similar issue where the output of ‘Decoder only model / GPT style’ implemented using the TransformerEncoder module.</p>
<p>Your intuition for the <code>-inf</code> is correct and if the rows of the output of the <code>attention_mask</code> is -inf, the model will return <code>nan</code> as output.</p>
<p>From what I understand, the solution to this would be to just use a right padding when preparing the inputs during validation as well.<br>
After this, create a causal mask like you usually would and then provide both of them to <code>self.decoder()</code>.</p>
Incorporating metadata into a resnet50 object detector

mycul: <p>Has anyone ever tried building something similar to some kind of Late Fusion with Resnet50 Object Detectors (Fast RCNN, Mask RCNN)?</p>
<p>Essentially have two models (one being resnet and the other being a smaller network that receives the metadata of the image which contains very important insights such as image size, empirical size of the image in inches, categorical info about the image).</p>
<p>All of this to hopefully have the model gain better insights somewhere down the line before making classifications.</p>
<p>Looking for advice and or resources !!</p>
<p>Grateful for any direction!</p>
Reaserch: 3D data and 2D discriminator

40uf411: <p>If I am working with 2D discriminator and 3D data, I would need to take slices from the three planes; my question is whether it is ok, to take random slices from the three planes, concatenate them and then pass them to the discriminator (knowing that some voxels might have more that one gradients in this case). Or is it better to do 3 separate discriminator passes and sum the losses?</p>
Using megacache vs saving /tmp/torchinductor_root

Fred_Liu: <p>Hi, I’ve been benchmarking the two different torch.compile caching strategies in this tutorial</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://docs.pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html">
  <header class="source">

      <a href="https://docs.pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html" target="_blank" rel="noopener nofollow ugc">docs.pytorch.org</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://docs.pytorch.org/tutorials/recipes/torch_compile_caching_tutorial.html" target="_blank" rel="noopener nofollow ugc">Compile Time Caching in torch.compile — PyTorch Tutorials 2.7.0+cu126...</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>When I was testing with flux, the compile time by loading in /tmp/torchinductor_root is around 20 seconds while the compile time using megacache and its api is 40 seconds. Without using any cache, compile time is around 170s. With other models like Llama, loading in /tmp/torchinductor_root always outperforms. I’m not sure of when megacache might be the better offering. What are the tradeoffs of using megacache vs saving torchinductor_root? The tutorial says megacache is portable and small, but I can achieve the same by compressing and archiving /tmp/torchinductor_root. Has anyone also gotten these results?</p>
<p>Here’s my torch.compile code:</p>
<pre><code class="lang-auto">    pipe = FluxPipeline.from_pretrained(
        MODEL_ID,
        torch_dtype=DTYPE,
    ).to(DEVICE)
    pipe.set_progress_bar_config(disable=True)
    if compile:
        pipe.transformer = torch.compile(
            pipe.transformer, mode="max-autotune-no-cudagraphs", dynamic=False
        )

        pipe.vae.decode = torch.compile(
            pipe.vae.decode, mode="max-autotune-no-cudagraphs", dynamic=False
        )
    return pipe</code></pre>
Inference time GPU memory management and gc

laoreja: <p>While measuring the GPU memory usage on inference time, we observe some inconsistent behavior: larger inputs end up with much smaller GPU memory usage through querying “nvidia-smi”.</p>
<p>We are wondering if the reason is that PyTorch is doing lazy garbage collection, i.e., the program only does garbage collection when there are not enough GPU memory left.</p>
<p>Also, actually, during inference time, the program can be super sparing with GPU memory if it keeps throwing away intermediate results. We are wondering if PyTorch has such implementation?</p>
<p>So many research papers’ reported inference time GPU memory usage is not that accurate?</p>

rasbt: <p>Remember to use</p>
<pre><code>model.eval()
with torch.set_grad_enabled(False):
    # your "inference" code here
</code></pre>
<p>otherwise your inference mode will create a huge graph in the background for autograd (which will keep growing and growing, since you don’t use .backward() during “inference”).</p>

laoreja: <p>Yes, we are using</p>
<pre><code class="lang-auto">with torch.no_grad()
</code></pre>
<p>for inference.</p>
<p>We are just wondering if reporting inference time GPU memory usage is not a good idea for showing your model’s memory efficiency based on the reasons stated above.</p>

rasbt: <p>Oh I see, sorry if I misunderstood. I don’t know about the exact automatisms, but I read somewhere that PyTorch uses as much memory as possible for computational throughput efficiency … So there might be some caching. There’s definitely some caching by cuDNN though, I noticed that when benchmarking. You can try setting</p>
<pre><code>torch.backends.cudnn.benchmark = True
</code></pre>
<p>and see if it makes some difference in your comparison.</p>

suke: <p>anybody know how i can release the gpu memory during inference?</p>
Docker images with different Python versions

caniko: <p>The tags in <a href="https://hub.docker.com/r/pytorch/pytorch/tags" rel="noopener nofollow ugc">Docker Hub Pytorch</a> are not explicit in their Python versioning. I assume they are all Python 3.7. Would it be possible to build images for every minor version from Python 3.7 and up? Not sure about the patches, build for every patch?</p>
<p>There is also a lack of explanatory text in the <a href="https://hub.docker.com/r/pytorch/pytorch/" rel="noopener nofollow ugc">main page</a>. Please add this also.</p>

marksaroufim: <p>Not ideal but the information is here <a href="https://github.com/pytorch/pytorch/blob/master/.circleci/docker/build.sh#L78" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/.circleci/docker/build.sh#L78</a></p>
<p>cc: <a class="mention" href="/u/malfet">@malfet</a></p>

barakugav: <p>As far as I understand it, i think it was moved to here <a href="https://github.com/pytorch/pytorch/blob/main/docker.Makefile" class="inline-onebox" rel="noopener nofollow ugc">pytorch/docker.Makefile at main · pytorch/pytorch · GitHub</a></p>
[DCP] how to load dcp ckpts?

GyLU: <p>Hi, I wonder how to load DCP ckpts for debugging, eg. similar to using <code>torch.load</code> for those non-DCP ckpts?</p>
<p>Currently, I have to define a model and an AppState to load DCP ckpts, which is too heavy for debugging.</p>

Hamza_Javaid: <p>You can use dcp.load_state_dict to load just the state dict without needing a full model:</p>
<pre><code class="lang-auto">import torch.distributed.checkpoint as dcp

# Load state dict directly
state_dict = {}
dcp.load_state_dict(
    state_dict=state_dict,
    storage_reader=dcp.FileSystemReader(checkpoint_dir),
)

# Now you can inspect the loaded tensors
for key, tensor in state_dict.items():
    print(f"{key}: {tensor.shape}")
</code></pre>

GyLU: <p>Hi, thanks for your answer! However, I found that in the latest doc, dcp.load_state_dict is deprecated and dcp.load should allocate tensor in state_dict <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=14" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

GyLU: <p>Oh, I think I find the correct way:</p>
<pre><code class="lang-auto">import torch.distributed.checkpoint as dcp
from torch.distributed.checkpoint import FileSystemReader

reader = FileSystemReader("/path/to/checkpoint")
metadata = reader.read_metadata()

sd_keys = set(key for key in metadata.state_dict_metadata.keys())

d = dcp.state_dict_loader._load_state_dict_from_keys(keys=sd_keys, storage_reader=reader)
print(d)
</code></pre>
How Can PyTorch Be Used to Enhance AI Models in Customer Support Systems Like Salesforce Agentforce?

anne_alice: <p>I’m currently exploring how AI can power smarter customer support systems, and during this journey, I came across the Salesforce Agentforce Specialist certification. It’s focused on applying AI within service workflows, like automated support, intelligent case routing, and real-time response generation. Since PyTorch is such a powerful framework for building NLP and conversational models, I started thinking: has anyone here worked on using PyTorch-based models (like BERT, DistilBERT, or even custom RNNs) in customer service contexts?</p>
<p>I’m especially curious if anyone has experience integrating PyTorch models into platforms like Salesforce or other CRMs, maybe for sentiment analysis, intent recognition, or chatbot responses. If you’ve done something similar, how did you manage deployment and inference? Did you face any performance or compatibility challenges?</p>
<p>While prepping for the Agentforce Specialist certification, I found Pass4Future practice questions pretty helpful. I’ve seen them mentioned on Quora and Reddit too. Has anyone else used them for AI or CRM-related certs? Curious if you found them useful or if there are other study tools you’d recommend that tie better into real ML use cases.</p>
<p>Looking forward to hearing your thoughts, especially if you’ve bridged the gap between PyTorch and real-world customer service applications!</p>

J_Johnson: <p>If you’re looking for an NLP application, I suggest starting with Huggingface. It’s built on PyTorch, but will reduce the learning curve and avoid having to reinvent the wheel.</p>
<p>In addition, you may find simply fine-tuning LoRA vectors over a SOTA model will cut the development bill quite a bit. There are many terrific LLMs there you can download and build on top of, depending on the size you want and application.</p>
<p>Huggingface Course: <a href="https://huggingface.co/learn/llm-course/chapter1/1" class="inline-onebox" rel="noopener nofollow ugc">Introduction - Hugging Face LLM Course</a></p>
<p>Fine-tuning Tutorial: <a href="https://huggingface.co/docs/transformers/v4.43.2/training" class="inline-onebox" rel="noopener nofollow ugc">Fine-tune a pretrained model</a></p>

brian_demajh: <p>I’ve done similar things many times.  An application I’m working quite a bit on lately involves building an extension of an LLM, rather than building an integration into a CRM.  The benefits include avoiding throwing out the baby with the bathwater: LLMs can already do quite a bit well and very often, they only need to be augmented with a point solution.</p>
<p>I describe the concept a bit more here:</p>
<p><a href="https://demajh.com/human_blog/ai_plugin.html" rel="noopener nofollow ugc">Ai Plugin Concept</a></p>
<p>Feel free to reach out if you have more questions!</p>
Does elastic torch support model parallelism

KevinL: <p>In the docs <a href="https://docs.pytorch.org/docs/stable/elastic/run.html" class="inline-onebox" rel="noopener nofollow ugc">torchrun (Elastic Launch) — PyTorch 2.7 documentation</a>, I only see it used with DDP. I wonder if anyone knows whether it also supports model parallelism. Thanks</p>

yf225: <p><a class="mention" href="/u/fegin">@fegin</a> curious is model parallelism supported in torchrun?</p>
FSDP2 and gradient w.r.t. inputs

ccasert: <p>I’m getting an error when trying to backprop the norm of the gradients of my model w.r.t. my inputs.</p>
<p>This throws errors such as</p>
<pre><code class="lang-auto">[rank0]: Traceback (most recent call last):
[rank0]:   File "....../test_autograd_fsdp/min_example.py", line 50, in &lt;module&gt;
[rank0]:     main()
[rank0]:   File "....../test_autograd_fsdp/min_example.py", line 46, in main
[rank0]:     loss.backward()
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/_tensor.py", line 648, in backward
[rank0]:     torch.autograd.backward(
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/autograd/__init__.py", line 347, in backward
[rank0]:     _engine_run_backward(
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/autograd/graph.py", line 823, in _engine_run_backward
[rank0]:     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: RuntimeError: setStorage: sizes [10, 100], strides [1, 10], storage offset 0, and itemsize 4 requiring a storage size of 4000 are out of bounds for storage of size 0
</code></pre>
<p>How can I get around this? Below is a minimal example. Thanks!</p>
<pre><code class="lang-auto">import os

import torch
import torch.nn as nn
import torch.distributed as dist
from torch.distributed.fsdp import fully_shard

class SimpleNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(10, 100)
        self.fc2 = nn.Linear(100, 100)
        self.fc3 = nn.Linear(100, 1)

    def forward(self, x):
        x = torch.relu(self.fc1(x))
        x = torch.relu(self.fc2(x))
        return self.fc3(x)

def main():
    dist.init_process_group(backend='nccl', init_method='env://')
    local_rank = int(os.environ["LOCAL_RANK"])
    torch.cuda.set_device(local_rank)
    device = torch.device("cuda", local_rank)

    model = SimpleNet().to(device)
    for name, module in model.named_modules():
        if module is not model and any(p.requires_grad for p in module.parameters(recurse=False)):
            fully_shard(module)
    fully_shard(model)

    x = torch.randn(32, 10, device=device, requires_grad=True)

    output = model(x)

    grads = torch.autograd.grad(
        outputs=output.sum(),
        inputs=x,
        create_graph=True,
        retain_graph=True
    )[0]


    loss = torch.norm(grads, p=2, dim=-1).mean()
    loss.backward()
    dist.destroy_process_group()

if __name__ == "__main__":
    main()
</code></pre>

yf225: <p>cc. <a class="mention" href="/u/weifengpy">@weifengpy</a> on FSDP2 related issue</p>
CUDA error: an illegal memory access was encountered problem

VIVEK_PANDEY: <p>Any way to debug this?<br>
RuntimeError: CUDA error: an illegal memory access was encountered<br>
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1<br>
Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>
<p>Pytorch:2.7.1, CUDA 11.8</p>

Hamza_Javaid: <p>I usually use cuda memcheck, you can use also</p>
<pre><code class="lang-auto">cuda-memcheck python your_script.py
</code></pre>
<p>Or enable sync reporting</p>
<pre><code class="lang-auto">export CUDA_LAUNCH_BLOCKING=1
export TORCH_USE_CUDA_DSA=1
export CUDA_DEVICE_WAITS_ON_EXCEPTION=1

# Then run your script
python your_script.py
</code></pre>

ptrblck: <p><code>cuda-memcheck</code> is deprecated so use <code>compute-sanitizer</code> and disable the caching allocator via: <code>PYTORCH_NO_CUDA_MEMORY_CACHING=1</code>.</p>
Is there any way to save multiple graphs from same model without redundantly saving their state dicts?

gabriel4321: <p>Hi, I want to save prefill / decode phase graphs of generative model (e.g., llama3.1) using <code>torch.export.save</code>. It looks like prefill / decode phase cannot be represented by one graph, so I called <code>torch.export.export</code> and <code>torch.export.save</code> twice each and this saves model’s state dict twice. I want to eliminate this redundancy. Is there any way to save model’s state dict once and share it between two exported graphs?</p>

heshuju: <p>I’m doing the same thing, do you have any solution?</p>

AnneStructo: <p>I think there is no any dirrect option provided by pytorch for duplicate the state_dict across multiple graphs. so this will export multiple phases of the same model causing unnecesary left overs.</p>
How to use torch with CUDA 12.9

Ahmed_Mhmouad: <p>I’m running with the following environment:</p>
<pre><code class="lang-auto">Windows 10
python 3.12.0
CUDA Version: 12.9  (according to `nvidia-smi`)
torch: 2.7.1
</code></pre>
<p>I did not find 12.9 at installation settings so i choose the nearest version 12.8<br>
I’m running the following code:</p>
<pre><code class="lang-auto">import torch
print(f"Version: {torch.__version__}, GPU: {torch.cuda.is_available()}, NUM_GPU: {torch.cuda.device_count()}")
</code></pre>
<p>And I’m getting:</p>
<pre><code class="lang-auto">Version: 2.7.1+cpu, GPU: False, NUM_GPU: 0
</code></pre>
<p>What is wrong ?</p>

ptrblck: <p>Your locally installed CUDA toolkit won’t be used as PyTorch ships with its own CUDA runtime dependencies.</p>
<p>Based on:</p>
<aside class="quote no-group" data-username="Ahmed_Mhmouad" data-post="1" data-topic="220827">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ahmed_mhmouad/48/77381_2.png" class="avatar"> Ahmed_Mhmouad:</div>
<blockquote>
<p><code>Version: 2.7.1+cpu, </code></p>
</blockquote>
</aside>
<p>You have installed the CPU-only binary and the installation log should have shown this.</p>

Ahmed_Mhmouad: <p>Oh thanks for replying, but i just used container and feel it easier.</p>

yogesh_jog: <p>Tried with below which is latest as of today and it works for me.</p>
<p>pip3 install torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/cu128" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu128</a></p>

HipHop_Delhi: <p>nope. this works pip3 install --pre torch torchvision torchaudio --index-url <a href="https://download.pytorch.org/whl/nightly/cu129" rel="noopener nofollow ugc">https://download.pytorch.org/whl/nightly/cu129</a></p>
Dendritic Learning: An open-source upgrade to PyTorch based on modern neuroscience

PerforatedAI: <p>We built this after studying recent neuroscience research showing that dendrites perform significant nonlinear computation that current AI completely ignores. Traditional artificial neurons are basically weighted sums + activation functions. Real neurons have dendrites that do complex processing before the cell body even sees the signal. Our implementation adds “dendritic support units” that can be dropped into existing PyTorch models with minimal code changes. This open source version focuses on gradient descent training, while we continue research on alternative training mechanisms for future releases.</p>
<p>Early results show models that can be up to 152x cheaper, 10x smaller, and 20% more accurate.</p>
<p><a href="https://github.com/PerforatedAI/PerforatedAI" rel="noopener nofollow ugc">Code</a></p>
<p><a href="https://www.perforatedai.com/case-studies" rel="noopener nofollow ugc">Results of our recent hackathon</a></p>
<p><a href="https://arxiv.org/pdf/2501.18018" rel="noopener nofollow ugc">Original Paper</a></p>
<p>Happy to answer questions about the implementation or share more benchmarks!</p>
ONNX dynamo export fails in 2.7 but worked in 2.3

vermont_waters: <p>I have a model which has a dictionary of torch tensor as inputs, E.g. with one element</p>
<p><code>export_input=({"head_1": torch.ones(2, 4)})</code></p>
<p>with torch 2.3 this could be exported as:</p>
<p><code>onnx_program = torch.onnx.dynamo_export(model_to_export, *export_input)</code></p>
<p>Now with torch 2.7, I am using the torch.onnx.export api. There is two things which dont work as expected:</p>
<ol>
<li>I have to use a sort of workaround for the input, since the API hardcodes non-tensor inputs into the model (not sure what that means). But if I just give it the input directly, it complains that the positional argument model_input_dict is not given. even though this should exactly be  <code>*export_input</code>.</li>
</ol>
<p>Hower I can solve it like this:</p>
<pre data-code-wrap="python"><code class="lang-python">        torch.onnx.export(
            model_to_export,
            args = (),
            kwargs={"model_input_dict": export_input[0]},
            f=model_path,
            dynamo=True)
</code></pre>
<ol start="2">
<li></li>
</ol>
<p>The export works, except for one line, where I put the up to here dynamic output into a static output:</p>
<pre><code class="lang-auto">    n_max = 1000  # fixed output size
    # class_all_ is of a dynamic size
    class_out = torch.zeros(
        n_max, dtype=class_all_.dtype, device=class_all_.device
    )
    class_out.index_copy_(0, indices, class_all_)
</code></pre>
<p>This worked fined in torch 2.3. Is this a bug or do I have to provide some other options?<br>
I get the following error which I cannot resolve:</p>
<pre><code class="lang-auto">&lt;class 'TypeError'&gt;: unsupported operand type(s) for *: 'int' and 'SymbolicDim'
⬆️
&lt;class 'torch.onnx._internal.exporter._errors.GraphConstructionError'&gt;: Error when calling function 'TracedOnnxFunction(&lt;function aten_index_put at 0x7575b2cee830&gt;)' with args '[SymbolicTensor(name='zeros', type=Tensor(FLOAT), shape=Shape([1000, 4]), producer='node_Expand_267', index=0), [SymbolicTensor(name='arange_2', type=Tensor(INT64), shape=Shape([SymbolicDim(u0)]), producer='node_Range_279', index=0)], SymbolicTensor(name='detach_2', type=Tensor(FLOAT), shape=Shape([SymbolicDim(u0), 4]), producer='node_Identity_254', index=0)]' and kwargs '{}'
⬆️
&lt;class 'torch.onnx._internal.exporter._errors.ConversionError'&gt;: Error when translating node %index_put : [num_users=1] = call_function[target=torch.ops.aten.index_put.default](args = (%zeros, [%arange_2], %detach_2), kwargs = {}). See the stack trace for more information.
</code></pre>
<p>In this error does the “*” stand for a multiplication? But the problem really just occurs in that index line</p>
Code crashing without error while using pytorch profiler with_stack=True

Deepak_Kumar1: <p>I am using window 11 with A10 GPU while running code with pytorch profiler with keeping with_stack=True<br>
my code snippet is as below:</p>
<p>with torch.no_grad():<br>
with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],profile_memory=True,record_shapes=True,with_stack=True,<br>
on_trace_ready=torch.profiler.tensorboard_trace_handler(f’./profiler_logs_proxylessNASGPUs8{self.group}')) as prof:<br>
model(data)</p>

Hamza_Javaid: <p>With Windows its a known issue.<br>
Maybe you can try nvidia nsys profiler<br>
<code>nsys profile -t cuda,cudnn,cublas python your_script.py</code></p>
<p>Also check you CUDA is compatible with your torch</p>

Deepak_Kumar1: <p>Thanks bro!! its working</p>
CUDA out of memory issues

VIVEK_PANDEY: <p>Is there a solution to this?</p>
<p>torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 263168.02 GiB. GPU 0 has a total capacity of 15.70 GiB of which 11.86 GiB is free. Including non-PyTorch memory, this process has 2.46 GiB memory in use. Of the allocated memory 2.21 GiB is allocated by PyTorch, and 21.10 MiB is reserved by PyTorch but unallocated.</p>
<p>P.S. CUDA Verion of GPU is 12.4 while my conda environment uses Pytorch 12.6 CUDA with CUDA toolkit 12.6</p>

Hamza_Javaid: <p>This OOM error shows that PyTorch is trying to allocate an impossibly large amount of memory (263,168 GB) on a GPU with only 15.7 GB capacity.</p>
<p>To find the exact issue, look at the line of code where the error occurs and check all tensor shape calculations leading up to it. The 263,168 GB allocation request strongly suggests something gone wrong with tensor dimensions.</p>

VIVEK_PANDEY: <p>I checked shapes of some important tensors, which seemed fine. I will check shape of every tensor and see.</p>

VIVEK_PANDEY: <p>Other machine with PyTorch CUDA 11.8 does not show this error, but the same PyTorch on my machine does. So, maybe its not Tensor size related.</p>