akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch.compile dynamic shapes in non-functional style

jjjj: <pre><code class="lang-auto">@torch.compile(backend="eager", fullgraph=True)
def accum(k, new_k) -&gt; torch.Tensor:
  if k.shape[2] &gt; 50:
    k = k[:, :, 1:, :]
  return torch.cat([k, new_k], axis=2)

k = torch.zeros((3, 3, 0, 3))
for i in range(1000):
  x = torch.zeros((3, 3, 1, 3))
  x[:] = i
  k = accum(k, x)
</code></pre>
<p>This functional-style code will work (it determines that dimension 2 of k is changing).<br>
but this</p>
<pre><code class="lang-auto">class Accum(torch.nn.Module):
  def __init__(self):
    super().__init__()
    self.register_buffer("k", torch.zeros(3, 3, 0, 3), persistent=False)
    torch._dynamo.mark_dynamic(self.k, 2)

  def forward(self, x):
    self.k = accum(self.k, x)
    return self.k


a = Accum()
a = torch.compile(a, backend="eager", fullgraph=True)
for i in range(1000):
  x = torch.zeros((3, 3, 1, 3))
  x[:] = i
  k = a(x)
</code></pre>
<p>will recompile each time through the loop until failing from too many recompiles because its guarding / specializing for the size of k<br>
tensor ‘L[‘self’]._buffers[‘k’]’ size mismatch at index 2. expected 0, actual 8</p>
<p>Is there a way I can make it behave like the functional style?</p>

jjjj: <p>I actually found that the second example can be made to work if you set.</p>
<p>torch._dynamo.config.force_nn_module_property_static_shapes = False<br>
torch._dynamo.config.force_parameter_static_shapes = False</p>
<p>Can’t figure out how I close this.</p>
Can I ask for feedback on a custom-built PyTorch (CUDA 12.8 / RTX 5070 Ti)?

gradiuse: <p>Hi, I built PyTorch manually from source using CUDA 12.8 and cuDNN 9.8 for compatibility with my RTX 5070 Ti.</p>
<p>I’d like to ask if it’s okay to share my custom <code>.whl</code> file and ask others (including PyTorch team) to test if it works well with SM120 architecture. I’m checking if my build is valid and stable enough.</p>
<p>Is it appropriate to post here for that kind of verification?</p>
<p>Thank you for your help!</p>
<p><a href="https://github.com/gradiuse/custom-pytorch-2.6.0a0" rel="noopener nofollow ugc">gradiuse/custom-pytorch-2.6.0a0: Custom-built PyTorch 2.6.0a0 for Windows 11 with CUDA 12.8 and cuDNN 9.8 - Successfully passed advanced stability tests - Supports: torch.compile, TF32, CUDA AMP - Status: Stable build (internal)</a></p>

ptrblck: <p>Why don’t you use the nightly binary with CUDA 12.8 instead?</p>

gradiuse: <p>Pytorch for gptsovits v3 doesn’t support graphics cards, so I made it before I knew the nightly build information. Thanks for letting me know. I’d like to write that I’ll download it right away and try it</p>
Installing PyTorch on a Grace Hopper GH200 node with GPU support

verdurin: <p>Having looked at some previous threads about this, I found that when using Python 3.11, without any extra specifications to pip, the installation of PyTorch with GPU support on a GH200 node would end up with PyTorch 2.0.1 with only CPU support.</p>
<p>To force CUDA 12.6 and Python 3.11, I had to specify:</p>
<pre><code class="lang-auto">pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu126 --python-version 311 --only-binary=:all: --target gh200-2025-02-18-3/lib64/python3.11/site-packages/
</code></pre>
<p>Just putting this here in case it’s of use to others, and perhaps if I’m doing this wrongly, someone who knows better can suggest an improvement.</p>

ptrblck: <p>Yes, to install the ARM+CUDA binaries, you should specify the <code>index-url</code> pointing to the latest CUDA runtime version we support.</p>

verdurin: <p>Sure - it was more the extra parameters I needed to add that I wanted to highlight, bearing in mind a previous comment of yours in a thread from last year that you were surprised pip hadn’t found the Python 3.11 wheels automatically.</p>

UnbearableFate: <p>Has PyTorch on the Grace Hopper GH200 been released? I haven’t found any other information besides this post.</p>

ptrblck: <p>Yes, you can install the nightly or latest stable releases for ARM + CUDA.</p>

youkaichao1: <p>it seems <code>pip install torch==2.6 --index-url https://download.pytorch.org/whl/cu126</code> works, but the wheels are not available in pypi index.</p>
How to build a one-to-many RNN model?

davidhkw: <p>Hi, I read the Pytorch RNN tutorial (<a href="https://pytorch.org/tutorials/intermediate/char_rnn_generation_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">NLP From Scratch: Generating Names with a Character-Level RNN — PyTorch Tutorials 2.6.0+cu124 documentation</a>) where it generates names (output: multiple characters) given a language (one input). It is done by a loop until the output is a EOS token. Are there ways to do it as in a many-to-one RNN model without a loop? Thanks</p>

J_Johnson: <p>Yes. Sort of. But then it’s no longer an RNN. It’s then called a Transformer. Please see this paper:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/1706.03762">
  <header class="source">
      <img src="https://arxiv.org/static/browse/0.3.4/images/icons/favicon-32x32.png" class="site-icon" width="32" height="32">

      <a href="https://arxiv.org/abs/1706.03762" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/402;"><img src="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png" class="thumbnail" width="500" height="500"></div>

<h3><a href="https://arxiv.org/abs/1706.03762" target="_blank" rel="noopener nofollow ugc">Attention Is All You Need</a></h3>

  <p>The dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Transformers work just fine asynchronously during training. This is done via a triangular matrix masking the latter tokens in the sequence. But during inference time, you still need to iterate.</p>
<p>Please see the tutorials starting here at Huggingface(library built on PyTorch):</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://huggingface.co/learn/nlp-course/en/chapter1/1">
  <header class="source">

      <a href="https://huggingface.co/learn/nlp-course/en/chapter1/1" target="_blank" rel="noopener nofollow ugc">huggingface.co</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/372;"><img src="https://huggingface.co/front/thumbnails/learn/nlp-course.png" class="thumbnail" width="690" height="372"></div>

<h3><a href="https://huggingface.co/learn/nlp-course/en/chapter1/1" target="_blank" rel="noopener nofollow ugc">Introduction - Hugging Face NLP Course</a></h3>

  <p>We’re on a journey to advance and democratize artificial intelligence through open source and open science.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Change CUDA Version in Colab or Kaggle

Firaki12345: <p>Hi, so i go to <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>
<p><strong>and choose my system requirements i.e the following:</strong></p>
<p><strong>Pytorch Build: Stable (1.10)</strong></p>
<p><strong>Linux</strong></p>
<p>Pip</p>
<p>Python</p>
<p><strong>CUDA Version: 11.3</strong></p>
<p><strong>The CUDA part can cause problems, as i run the following cmd on both kaggle and colab and get different CUDA version.</strong></p>
<p><strong>Kaggle</strong></p>
<p><code>!nvcc --version</code></p>
<p><code>nvcc: NVIDIA (R) Cuda compiler driver</code></p>
<p><code>Copyright (c) 2005-2020 NVIDIA Corporation</code></p>
<p><code>Built on Wed_Jul_22_19:09:09_PDT_2020</code></p>
<p><code>Cuda compilation tools, release 11.0, V11.0.221</code></p>
<p><code>Build cuda_11.0_bu.TC445_37.28845127_0</code></p>
<p><strong>Google Colab :</strong></p>
<p><code>nvcc: NVIDIA (R) Cuda compiler driver</code></p>
<p><code>Copyright (c) 2005-2020 NVIDIA Corporation</code></p>
<p><code>Built on Mon_Oct_12_20:09:46_PDT_2020</code></p>
<p><code>Cuda compilation tools, release 11.1, V11.1.105</code></p>
<p><code>Build cuda_11.1.TC455_06.29190527_0</code></p>
<p><strong>So how can i change the CUDA Version to match 11.3 ? Or will it mess up the Notebooks environment. Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></strong></p>

ptrblck: <p>You won’t be able to change the local CUDA toolkit easily. The PyTorch binaries ship with their own CUDA runtime so unless you are building PyTorch from source or a custom CUDA extension the local CUDA toolkit won’t be used.</p>

Doug_Chang: <p>This is easy to do. I just had this problem.</p>
<ol>
<li>lsb_release -a to find ubuntu version in colab</li>
<li><a href="http://developer.nvidia.com/cuda-toolkit-archive" class="inline-onebox" rel="noopener nofollow ugc">CUDA Toolkit Archive | NVIDIA Developer</a> to find cuda version and OS version you want</li>
<li>network install follow instructions</li>
<li>remove and create a new symlink for cuda or manage environment according to your needs.</li>
</ol>
No CUDA GPUs available

Nachofc333: <p>When i try to run my code, i get this message:<br>
torch._C._cuda_init()<br>
RuntimeError: No CUDA GPUs are available</p>
<p>But if i run this command it looks like it is detecting it:<br>
python -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.device_count()); print(torch.cuda.get_device_name(0))</p>
<p>output:<br>
True<br>
1<br>
NVIDIA GeForce RTX 4060 Ti</p>
<p>And with the command nvidia-smi:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8a00884314c08adaa6c2fd551945ec1eec4d093.png" data-download-href="https://discuss.pytorch.org/uploads/default/a8a00884314c08adaa6c2fd551945ec1eec4d093" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/8/a8a00884314c08adaa6c2fd551945ec1eec4d093.png" alt="image" data-base62-sha1="o3J5fHgIkstDV1jL4I7Tl5m7H2z" width="631" height="338"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">631×338 7.33 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
i am using WSL2 ubuntu. Don´t know which is the problem. It looks like pytorch is detecting my gpu, but quen i try and run the code it fails</p>

ptrblck: <p>Based on the error message it seems PyTorch is able to detect your GPU via <code>nvml</code> (which is also used by <code>nvidia-smi</code>) but fails to initialize it which often points to a driver or setup issue.</p>
Torch.load and mmap

yanfeiwong: <p>I have a huge data, lets say a large cache of vae encoded stuff, I want to access the values I need, but after that I do not need to keep them in my cpu memory, how do I do that?<br>
load is simple:</p>
<p>data = torch.load(r"huge_vae_dict.ckpt",weights_only=1,mmap=1)</p>
<p>but once I accessed all the value, the whole dict ends up in my ram</p>
<p>with torch.no_grad():<br>
for k in tqdm(data):<br>
v = data[k]+1</p>
<p>I wish we can access the data but not end with with them all in the memory.</p>

claudia: <p>I have been working on this problem recently as well. I just found the following library. Are there any other similar libraries?</p><aside class="onebox githubrepo" data-onebox-src="https://github.com/hnmr293/ckpt-streamer">
  <header class="source">

      <a href="https://github.com/hnmr293/ckpt-streamer" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <div class="github-row" data-github-private-repo="false">
  <img width="690" height="344" src="https://opengraph.githubassets.com/7cdb536077679579cede99889d3f4a7f/hnmr293/ckpt-streamer" class="thumbnail">

  <h3><a href="https://github.com/hnmr293/ckpt-streamer" target="_blank" rel="noopener nofollow ugc">GitHub - hnmr293/ckpt-streamer: streaming ckpt loader for low RAM people</a></h3>

    <p><span class="github-repo-description">streaming ckpt loader for low RAM people</span></p>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Finnetuning problems on googlenet architecture

Gabriel_Duran: <p>Hello guys, Im trying to make finnetuning with a googlenet pretrained model. I only want to modify the fully connected layers. For that reason, I tried to extract the feature part without the fully connected layers, the avgpool and flatten functions from the pretrained model, but I got an error.</p>
<pre data-code-wrap="shell"><code class="lang-shell">AttributeError: 'GoogLeNet' object has no attribute 'features'
</code></pre>
<p>Do you know how to extract the convolutional part from the google net architecture or similar architectures without the feature object?</p>
<p>I saw one form used in Resnet 18 from <a href="https://github.com/ar-ambuj23/covid19_pocus_ultrasound_pytorch/blob/main/code/models/resnet.py" class="inline-onebox" rel="noopener nofollow ugc">covid19_pocus_ultrasound_pytorch/code/models/resnet.py at main · ar-ambuj23/covid19_pocus_ultrasound_pytorch · GitHub</a></p>
<p>And he used:</p>
<pre><code class="lang-auto">self.model = nn.Sequential(*list(self.model.children())[:-1])
</code></pre>
<p>The code that I used is</p>
<pre><code class="lang-auto">        self.model = models.googlenet(pretrained=True)

        # freeze weights of base model except last cnn layer
        # model.parameters() does not include max pooling layers
        for param in self.model.parameters():
            param.requires_grad = False
                
        # Taking only sequential part
        self.model = self.model.features
</code></pre>
<p>And is addapted from <a href="https://github.com/ar-ambuj23/covid19_pocus_ultrasound_pytorch/blob/main/code/models/vgg.py" class="inline-onebox" rel="noopener nofollow ugc">covid19_pocus_ultrasound_pytorch/code/models/vgg.py at main · ar-ambuj23/covid19_pocus_ultrasound_pytorch · GitHub</a></p>

ptrblck: <p><code>GoogleNet</code> uses <code>self.fc</code> to access the last linear layer as seen <a href="https://github.com/pytorch/vision/blob/8ea4772e97bc11b2cfee48a415e7df8cd17fa682/torchvision/models/googlenet.py#L92">here</a>.</p>
DDP Training Hangs after completing Epoch

Abhinav_Rai: <p>I’ve been stuck on this issue for some time and need some guidance.</p>
<p>My network contains a moduledict, where each key refers to a city and the value is a classifier for that city. Depending on the data input to the model, different classifiers are called and trained. This model works fine on a single-gpu setting. However, when I switch to multi GPUs the training hangs at the end of the epoch. My guess is this is due to the fact that based on the data present in the batch different classifiers are being called in different GPUs(this I know is happening). Hence, since the network graph is different across the GPUs the model is not able synchronize properly. Is my assumption here correct ? If so why does it hang only at the end of the epoch and what can I do to fix this issue.</p>
<p>Thanks</p>

yf225: <p>There are allreduce operations at end of DDP, which could be where the hangs show up.</p>
<p>NCCL hangs usually happen because of one of the following reasons:</p>
<ol>
<li>One of the ranks is taking too long to finish, and other ranks timed out waiting for that rank.</li>
<li>One of the ranks crashed, and other ranks timed out waiting for that rank.</li>
<li>One of the ranks has a classifier or something else that issues an extra collective op, and there is mismatch on the global order of collective ops across all ranks (i.e. deadlock).</li>
</ol>

ptrblck: <p>Besides the good explanations shared by <a class="mention" href="/u/yf225">@yf225</a> you could also rerun your code with <code>NCCL_DEBUG=INFO</code> to get more additional information about potential issues.</p>
How to identify tensors in autograd saved tensors hook?

mseeger: <p>I learned about <a href="https://pytorch.org/tutorials/intermediate/autograd_saved_tensors_hooks_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Hooks for autograd saved tensors — PyTorch Tutorials 2.6.0+cu124 documentation</a>.</p>
<p>But I am missing one detail. I’d like to implement a pack/unpack hook which applies to specific nodes only. I’d expect that when the tensor is created in forward, I can tag it somehow and then identify it in <code>pack_hook</code>. But I failed to find anything.</p>
<p>What is the simplest way to identify specific tensors in the pack hook?</p>

mseeger: <p>My first thought was to use <code>grad_fn.name</code> to identify the tensors, also to tag some extra information that I need in the <code>pack_hook</code>. But I can only access this once the node is created, and this means <code>pack_hook</code> will be called already, right?</p>

mseeger: <p>I guess I found a solution to this, but it is pretty indirect. I push the annotation into a list and check the list in the hook, popping the annotation if there is one.</p>
In-place copy of tensors sharing the same underlying storage

jjjj: <pre><code class="lang-auto">x = torch.randn(size=(10, 10))
x[:, 0:8] = x[:, 1:9]
x[0:8, :] = x[1:9, :]
</code></pre>
<p>In pytorch (2.6) the second assignment will error with:</p>
<p>RuntimeError: unsupported operation: some elements of the input tensor and the written-to tensor refer to a single memory location. Please clone() the tensor before performing the operation.</p>
<p>But I think the first assignment is also unsafe and can result in undefined behavior? Why does it not error except on the first dimension or is there some reason that it is actually safe?</p>