akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Where can I find a version of pytorch which is suitable for CUDA12.8?

Nicholas: <p>My os is win11 and gpu is 5090. First I tried CUDA12.8 but I can’t find a suitable version of pytorch so I have to download CUDA11.8 and use an old version of pytorch.<br>
However, when I tried to train my network, it shows that my gpu with CUDA capability sm_120 is not compatible with the pytorch version.<br>
I wonder where can I find a version of pytorch which is suitable for CUDA12.8?<br>
Thanks!</p>

ptrblck: <aside class="quote" data-post="13" data-topic="216099">
  <div class="title">
    <div class="quote-controls"></div>
    <img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/pytorch-support-for-sm120/216099/13">Pytorch support for sm120</a> <a class="badge-category__wrapper " href="/c/deployment/14"><span data-category-id="14" style="--category-badge-color: #BF1E2E; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="A category of posts focused on production usage of PyTorch. Mobile deployment is out of scope for this category (for now…  )"><span class="badge-category__name">deployment</span></span></a>
  </div>
  <blockquote>
    The nightly wheels for Windows are available as of now, e.g. for Python==3.12 you can find the download <a href="https://download.pytorch.org/whl/nightly/cu128/torch-2.7.0.dev20250220%2Bcu128-cp312-cp312-win_amd64.whl#sha256=d91399263c6d59f0258ad586ad887adbd7e94ba2fe1c9e395dfda48249345ae1">here</a>.
  </blockquote>
</aside>


Nicholas: <p>Thanks! And according to the other replies, I wonder if torchvision and other libraries will support CUDA12.8 currently? Thanks!</p>

long_yi: <p>I’m having the same issue and it hasn’t been resolved after following the reply above</p>
Warmup on a side stream

dskhudia: <p>Both the docs and blog mention that for graph capture warmup must happen on a side stream. Any reason why?</p>
<p>“Warmup must occur on a side stream”</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/">
  <header class="source">
      <img src="https://pytorch.org/favicon.ico?" class="site-icon" width="32" height="32">

      <a href="https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/" target="_blank" rel="noopener nofollow ugc">PyTorch</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/297;"><img src="https://pytorch.org/assets/images/cudagraphs-pytorch.png" class="thumbnail" width="690" height="297"></div>

<h3><a href="https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/" target="_blank" rel="noopener nofollow ugc">Accelerating PyTorch with CUDA Graphs</a></h3>

  <p>Today, we are pleased to announce a new advanced CUDA feature, CUDA Graphs, has been brought to PyTorch. Modern DL frameworks have complicated software stacks that incur significant overheads associated with the submission of each operation to the...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://pytorch.org/docs/stable/notes/cuda.html">
  <header class="source">

      <a href="https://pytorch.org/docs/stable/notes/cuda.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://pytorch.org/docs/stable/notes/cuda.html" target="_blank" rel="noopener nofollow ugc">CUDA semantics — PyTorch 2.6 documentation</a></h3>

  <p>A guide to torch.cuda, a PyTorch module to run CUDA operations</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
How to get GPU running in Raspeberry Pi 4 for training?

Evandro_Raphaloski: <p>Hello,</p>
<p>Please I would like to know if is it possible to run Pytorch CNN models on GPU in Raspberry Pi 4? I saw an inference example, but I need for training. I install the torch through pip, and it just get CPU for execution.</p>
<p>Is there any solution for use the GPU in Raspberry Pi 4? Thanks.</p>
Torch.distributed.checkpoint.save hangs while writing the .metadata file

eminorhan: <p>While saving a distributed checkpoint to disk with <code>torch.distributed.checkpoint.save</code> (across 512 ranks), the model parts are successfully saved in <code>__x_y.distcp</code> files, but occasionally the process group hangs while writing the <code>.metadata</code> file and times out, causing the whole job to fail. The part of the code that saves the checkpoint is straightforward and looks something like this (simplifying):</p>
<pre data-code-wrap="python"><code class="lang-python">import torch.distributed as dist
import torch.distributed.checkpoint as dcp

pg = dist.new_group(backend="gloo")
...
dcp.save(states, checkpoint_id=checkpoint_id, process_group=pg)
</code></pre>
<p>Exact replication may be difficult, since I’m using MI250X GPUs (ugh) and even on my end the failure is stochastic: maybe half the time <code>dcp.save()</code> succeeds without an issue and half the time the process group hangs as described above.</p>
<p>I’m just trying to understand why the <code>.distcp</code> files are always successfully saved, and the process group seems to hang only when writing the <code>.metadata</code> file. Any pointers and suggestions for potential workarounds would be much appreciated.</p>
<p>My <code>torch</code> version is <code>2.6.0.dev20241005+rocm6.2</code>, for what it’s worth.</p>

c-p-i-o: <p><code>dist.new_group(backend="gloo")</code> creates a connection mesh between the ranks.<br>
Do you know what transport is being used by <code>gloo</code> under the covers for the 2 cases?<br>
Is it TCP or RDMA or both?<br>
Gloo has support for both.</p>

fngarrett: <p>You said that when it hangs, you get a <code>TimeoutError</code>. Would you be able to post the full output when this occurs?</p>
<p>I am also curious about the consistency of <code>TimeoutError</code>’s if you were to just “try again” for the same job.</p>
<p>There’s likely a better implementation, but something like this:</p>
<pre data-code-wrap="python"><code class="lang-python">try:
    dcp.save(states, checkpoint_id=checkpoint_id, process_group=pg)
except TimeoutError:
   dcp.save(states, checkpoint_id=checkpoint_id, process_group=pg)
</code></pre>
<p>Also, I am currently looking around <a href="https://github.com/pytorch/pytorch/blob/976ff5cf01bb1aaa813df64c42b73b88cc3b9062/torch/distributed/checkpoint/state_dict_saver.py#L3350" rel="noopener nofollow ugc">here</a> in the source code, since it just fails at writing metadata.</p>
Cant't find amdgpu.ids when running in venv

Outssiss: <p>Hey everyone. I recently swapped to arch and I’ve been running into this issue where torch is not capable of finding the amdgpu.ids file.</p>
<p>I installed python via pyenv (tried both 3.11.11 and 3.12.8), created a venv and then installed torch in that virtual environment, this throws the warning “amdgpu.ids: No such file or directory” two times in a row, I assume one for the integrated GPU and another one for the dedicated one.</p>
<p>But if instead I install torch directly on the global pyenv install this warning goes away and everything works as expected.</p>
<p>Some information about the machine:</p>
<ul>
<li>CPU: Ryzen 7 7700X</li>
<li>GPU: Radeon RX 7800 XT</li>
<li>ROCm version: 6.2</li>
</ul>
<p>I can see that when installing on the venv, the amdgpu.ids file is still present at:<br>
/home/josef/CodeThings/PytorchThingsEnv/.venv/lib/python3.11/site-packages/torch/share/libdrm/amdgpu.ids</p>
<p>And when installed on the global install, it’s located at:<br>
/home/josef/.pyenv/versions/3.11.11/lib/python3.11/site-packages/torch/share/libdrm/amdgpu.ids</p>
<p>Updated to the issue: I tried now installing torch in the global instance and then also in the venv and noticed that this got rid of the warnings when running in the venv. Turns out that torch in the venv still looks for the  amdgpu.ids file in the location of the global install (/home/josef/.pyenv/versions/3.11.11/lib/python3.11/site-packages/torch/share/libdrm/amdgpu.ids) instead that on the venv location (/home/josef/CodeThings/PytorchThingsEnv/.venv/lib/python3.11/site-packages/torch/share/libdrm/amdgpu.ids) and if I remove the file from the global location, the warnings come back.</p>

fngarrett: <p>Could you share the outputs of <code>which python</code>, <code>which pip</code>, and <code>pip freeze</code> in the virtual environment as well as in the global pyenv?</p>
Is Pytorch 1.10.0 for Rocm 4.2 still supported?

outdatedhardware: <p>I have RX 570 graphics card, which has Polaris architecture, according to techpowerup, the last version of Rocm to support this card is Rocm 4.4.</p>
<p>But according to this <a href="https://www.youtube.com/watch?v=lCOk6Id2oRE" rel="noopener nofollow ugc">video</a>, the last version of Rocm to support this card is Rocm 5.7.3.</p>
<p>I’m trying to run stable diffusion on Ubuntu 22.04, I first tried with Rocm 5.7.3 but I got segmentation fault errors. So I uninstalled Rocm 5.7.3 and installed the older version but when I try to install Pytorch 1.10.0 for Rocm 4.2 or Rocm 4.1 or Rocm 4.0.1 as mentioned on this link: <a href="https://pytorch.org/get-started/previous-versions/" class="inline-onebox" rel="noopener nofollow ugc">Previous PyTorch Versions | PyTorch</a></p>
<p>Using these commands I’m getting long verbose version mismatch errors</p>
<blockquote>
<h1>ROCM 4.2 (Linux only)</h1>
<p>pip install torch==1.10.1+rocm4.2 torchvision==0.11.2+rocm4.2 torchaudio==0.10.1 -f</p>
<h1>ROCM 4.1 (Linux only)</h1>
<p>pip install torch==1.10.1+rocm4.1 torchvision==0.11.2+rocm4.1 torchaudio==0.10.1 -f</p>
<h1>ROCM 4.0.1 (Linux only)</h1>
<p>pip install torch==1.10.1+rocm4.0.1 torchvision==0.10.2+rocm4.0.1 torchaudio==0.10.1 -f</p>
</blockquote>
<p>Removed pytorch links from blockquote because new users are only allowed to post 2 links.</p>

fngarrett: <p>It seems that PyTorch 1.10.0, 1.10.1, 1.10.2 + rocm4.2 only has wheels for Python versions 3.6, 3.7, 3.8, 3.9 (list of wheels <a href="https://download.pytorch.org/whl/torch/" rel="noopener nofollow ugc">here</a>). Could you verify that you are trying to install into an environment that’s using one of these Python versions?</p>
Fixing seeds affects performance?

iliasslasri: <p>Hey all,<br>
Referring to topics like <a href="https://discuss.pytorch.org/t/how-could-i-fix-the-random-seed-absolutely/45515">How could I fix the random seed absolutely</a>. I couldn’t find any resources talking about the effects of fixing all seeds and going<code>torch.backends.cudnn.enabled=False torch.backends.cudnn.deterministic=True</code>. Does this affect performance of training at all ?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import numpy as np
import random
import my_model
import my_dataset

torch.manual_seed(123)
torch.cuda.manual_seed(123)
np.ranom.seed(123)
random.seed(123)
torch.backends.cudnn.enabled=False
torch.backends.cudnn.deterministic=True

def train():
...

if __name__ == '__main__':
    train()
</code></pre>

ptrblck: <aside class="quote no-group" data-username="iliasslasri" data-post="1" data-topic="217176">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/67e7ee/48.png" class="avatar"> iliasslasri:</div>
<blockquote>
<p>Does this affect performance of training at all ?</p>
</blockquote>
</aside>
<p>I assume performance means training speed here. Yes, disabling cuDNN or selecting deterministic algorithms could slow down your code.</p>
How to implement gradient clipping in FSDP2 (fully_shard)

ivan.prado: <p><code>FullyShardedDataParallel</code> implements the method <a href="https://pytorch.org/docs/stable/fsdp.html#torch.distributed.fsdp.FullyShardedDataParallel.clip_grad_norm_" rel="noopener nofollow ugc">clip_grad_norm_</a>, but what would the equivalent for FSDP2 (fully_shard)? If there is no method, how could it be implemented?</p>
<p>Thank you</p>
What does the learning rate do in LBFGS?

dherrera1911: <p>My understanding of the LBFGS algorithm is that it updates the parameters by using a line-search algorithm. That is in opposition to using a fixed-size update given by the learning rate, which is what most other optimizers do. However, LBFGS has an <code>lr</code> parameter, like othe roptimizers. I am wondering if someone knows what role precisely <code>lr</code> plays in the LBFGS optimizer.</p>
Cuda 12.4 vs Pytorch ___ compatibility (latest)

Mohan_Krishna: <p>For Cuda 11.8   =&gt;   * <strong>PyTorch 1.13.0 to 2.0.1</strong>   are compatible.<br>
What about Cuda 12.4 =&gt; Which pytorch latest versions are available?</p>

ptrblck: <p>You can check previous releases <a href="https://pytorch.org/get-started/previous-versions/">here</a>.</p>
Batch first gru with 2 layers and bidirectional is allergic to batch size > 1

edmondj: <p>Why does this print 2 different values ?</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

class Test(nn.Module):
    def __init__(self):
        super().__init__()
        self.lin3 = nn.Linear(5, 400)
        self.rnn2 = nn.GRU(1, 100, num_layers=2, bidirectional=True, batch_first=True)

    def forward(self, discrete_code):
        c2 = self.lin3(discrete_code/5).reshape((4, discrete_code.shape[0], 100))
        x, _ = self.rnn2(torch.zeros(discrete_code.shape[0], 64, 1).to(device), c2)
        return x[:1, 0, 0]

model = Test().to(device)
candidats = torch.randn(10, 5).to(device)
print(model(candidats[:1]))  # e.g., tensor([0.1234])
print(model(candidats[:2]))  # e.g., tensor([0.1234]) EXPECTED SAME
</code></pre>
Question about graph CNNs and pooling

flf: <p>Hi,</p>
<p>Hopefully this is the right place to post this question.</p>
<p>I am trying to get some predictions on some graph data using torch geometric; the graph represents a 2D finite element mesh with n nodal features, and the geometry has been normalized so that it fits the [0, 1] square. The predictions would be for simple linear PDEs on that mesh. Each training/test sample consists of a different geometry.</p>
<p>The best effort so far is to fit the graph in a set of images size x size x n and run a standard CNN on it, i.e. not actually using torch geometric. The CNN compresses the features from e.g. 30 x 30 x 3 into e.g. 4 x 4 x 64, then flatten and a fully connected layer.</p>
<p>I am trying to achieve similar performances by using the original graph and running some units of the sequence GCN + InstanceNorm + ReLU + MLP + TopKPooling, the latter to a fixed number. When the size of the graph is down to a reasonable amount, then flatten and a fully connected layer.</p>
<p>With the latter configuration I am not achieving the same results as with the CNN (not even close). I imagine it has to do with the CNN “having encoded” the geometric information of the system in its rigid structure, whilst the graph CNN may not be able to capture this. I was wondering what your opinion on this is, and if you had any suggestions. Please let me know if you need further info, my description so far may have been a tad too short.</p>
<p>Thanks!<br>
Francesc.</p>
cuDNN error: CUDNN_STATUS_EXECUTION_FAILED [use L40s]

ckku841023: <p>I have a program for model training and inference that relies on many customized functions, which must be executed in a pytorch1.5.0+cu9.2 environment. I set up an Ubuntu 18.04 environment in the cloud with an L40s GPU. Currently, I installed the CUDA driver at version 12.4. When I run the program, I get the following error message:</p>
<pre><code class="lang-auto">return torch.batch_norm(   
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
</code></pre>
<p>So far, I have tried several approaches, including:</p>
<ol>
<li>
<p>Downloading a lower version of the runtime toolkit (e.g., toolkit 10.2), but it still reports the above error. <em>(P.S. I haven’t tried downloading the CUDA toolkit 9.2 because the official toolkit only supports up to Ubuntu 16.04.)</em></p>
</li>
<li>
<p>I found online that someone mentioned the issue might be due to different CUDA driver versions and that the toolkit library (cuDNN) is dynamically linked. I tried adjusting the linking order, but still encountered the same error.</p>
</li>
<li>
<p>Someone online mentioned that the L40s only supports CUDA 11.x or later. Could this be the reason why I cannot run pytorch1.5.0+cu9.2 on the L40s? I did try upgrading my program to run successfully on pytorch1.8.0+cu111, but since the program is tightly coupled with low-level hardware-specific syntax, I cannot upgrade it without permission. I would like to find an alternative solution that allows me to use the L40s to train models under pytorch1.5.0+cu9.2.</p>
</li>
</ol>

ptrblck: <p>Your PyTorch version is too old to execute code on your L40 device. Update PyTorch to any current version with CUDA 11 or 12.</p>

ckku841023: <p>Now, I confirm that the version before CUDA 11 cannot run on the L40s graphics card. Thanks alot  for your suggestion.</p>
FSDP without data parallelism

Yash_Lal: <p>Is there a way to use the FSDP system without using data parallelism. The tutorial here (<a href="https://pytorch.org/tutorials/intermediate/FSDP_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Fully Sharded Data Parallel(FSDP) — PyTorch Tutorials 2.2.1+cu121 documentation</a>) seems to use both. In my use case, I want the benefit of splitting up large gradients in backprop across multiple GPUs but I don’t want to spawn multiple training processes simultaneously. Is this possible?</p>

luigiscorzato: <p>From my understanding it should be possible (I haven’t tried), but then you would be using the additional GPUs (all GPUs but one) just as a storage because your gradient doesn’t fit in memory, right. AM I wrong?</p>

Yash_Lal: <p>Yes that is the idea, the gradients are too large and I want more storage for them by combining GPUs.</p>

agu: <p>The FSDP implementation assumes that each process uses its own GPU, so if you are not spawning multiple processes, then FSDP probably will not work.</p>

MADHAV_THAKKER: <p>Were you able to get more clarity how to do this?</p>

Yash_Lal: <p>Not really, I ended up abandoning the FSDP approach and resolving the memory issues a different way by handling my data differently.</p>

MADHAV_THAKKER: <p>Yeah. My use case is also very similar and FSDP seems very confusing to get into. What did you end up doing?</p>

Yash_Lal: <p>I was using very large graphs with PytorchGeometric, but for my use case I was able to split the graphs in half which helped. We also found better performance with smaller models which reduced the computational burden further. This observation also meant that we no longer trying the larger models causing problems because we concluded that smaller models were doing better.</p>

MADHAV_THAKKER: <p>Great. Thank you so much for your help.</p>
DataLoader batch_size in DDP for multi-gpu and multi-node

Joeyoung: <p>Hi,</p>
<p>I want to set the batch size to 128.<br>
When I use 8 gpus ( 4 gpus / node and 2 nodes ) - so world_size=8, ngpus_per_node=4 -<br>
what number do I need to set to batch_size= in DataLoader?<br>
(I run torchrun with DDP.)</p>
<p>128/world_size = 16 or 128/ngpus_per_node = 32 or 128 ?</p>
<p>Thank you!</p>
PyTorch 2.5.1 -fstack-protector-strong does not result in stack-protected libpytorch.so

Steve_Taylor: <p>Ubuntu 18.04<br>
gcc 9.4.0<br>
ldd 2.27<br>
Python 3.11</p>
<p>I am attempting to build PyTorch 2.5.1 so that the resulting shared libraries are “hardened”, such that when we run <code>hardening-check</code> from Debian’s <code>devscripts</code> against them, it reports that stack protection is enabled.</p>
<p>I’ve added “-fstack-protector-strong” to the <code>extra_compile_args</code> in setup.py, and all but two libraries (libtorch.so and libtorch_global_deps.so) are successfully hardened.</p>
<p>Nothing I have tried so far has had any effect on these two libraries. Does anyone know what I’m doing wrong?</p>
<p>I notice that these libraries are very small compared to the others that get produced during the build.  Are these libraries special somehow?</p>

Steve_Taylor: <p>Solved!  <code>libtorch.so</code> and <code>libtorch_global_deps.so</code> are empty helper libraries, which is explained via comments in <code>CMakeLists.txt</code>.</p>
<p>I gather this means that there isn’t a stack to protect, so the compiler flag has no effect.  Which rhymes, so it must be true.</p>
Torch version + cuda stable compatibility details:

Mohan_Krishna: <p>I have installed  NVIDIA-SMI 550.120                Driver Version: 550.120        CUDA Version: 12.4</p>
<p>$ nvcc --version<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2023 NVIDIA Corporation<br>
Built on Fri_Jan__6_16:45:21_PST_2023<br>
Cuda compilation tools, <strong>release 12.0, V12.0.140</strong><br>
Build cuda_12.0.r12.0/compiler.32267302_0<br>
I want a stable versions of torch and cuda compatibility for NVIDIA RTX A5000</p>
<p>which version should i proceed on cuda toolkit, torch, driver</p>

ptrblck: <p>Your locally installed CUDA toolkit won’t be used as PyTorch binaries ship with their own CUDA runtime dependencies. You would need to install an NVIDIA driver first and can install any binaries afterwards. To install the latest stable version with CUDA support run <code>pip install torch</code>.</p>

Mohan_Krishna: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/0/70fc50686d343a081b10ba8fe854e4c251ab3b16.png" data-download-href="https://discuss.pytorch.org/uploads/default/70fc50686d343a081b10ba8fe854e4c251ab3b16" title="Screenshot from 2025-02-25 20-15-49"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/0/70fc50686d343a081b10ba8fe854e4c251ab3b16.png" alt="Screenshot from 2025-02-25 20-15-49" data-base62-sha1="g7w3xEC9UiS9vfHViW8oHXEo9bE" width="690" height="346" data-dominant-color="212323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2025-02-25 20-15-49</span><span class="informations">907×455 38.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Hiring: ML Engineers @ Red Hat

mejones_redhat: <p>Red Hat AI ENG is looking for ML Engineers to join our team and drive Red Hat’s contribution to key open source AI communities. We work on PyTorch, vLLM, Triton and their many dependent and related projects. Our mission is to support and foster these communities, as well as productize them as part of our RHEL AI and OpenShift AI products. We are hiring all levels of ML Engineers from early career to Senior Principal, as well as looking for people managers for these teams (preferred locations for these roles are Raleigh, NC and Boston, MA). Send your resume to <a href="mailto:mejones@redhat.com">mejones@redhat.com</a> if you are a qualified applicant and interested in learning more.</p>
Running out of GPU memory when loading a huggingface model

chrathans: <p>I am trying to load a hugging-face model <code>microsoft/Phi-3-medium-4k-instruct</code> to my <code>GPU</code>. The model size (26 GB) is bigger than VRAM (24 GB), but I do have 2 GPUs. Can I load my model using both GPUs (in batches) without facing this issue?</p>
<p>The code I am using is the following:</p>
<pre><code class="lang-auto">import os
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
import deepspeed
import time
import pdb

tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-medium-4k-instruct", use_fast=True)
model = AutoModelForCausalLM.from_pretrained("microsoft/Phi-3-medium-4k-instruct", torch_dtype=torch.float16, low_cpu_mem_usage=True,device_map=local_rank)
pdb.set_trace()
</code></pre>

anantguptadbl: <p><a class="mention" href="/u/chrathans">@chrathans</a></p>
<p>Can you try to create the device_map at the layers level</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://huggingface.co/docs/accelerate/v0.25.0/en/concept_guides/big_model_inference">
  <header class="source">

      <a href="https://huggingface.co/docs/accelerate/v0.25.0/en/concept_guides/big_model_inference" target="_blank" rel="noopener nofollow ugc">huggingface.co</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/372;"><img src="https://huggingface.co/front/thumbnails/docs/accelerate.png" class="thumbnail" width="690" height="372"></div>

<h3><a href="https://huggingface.co/docs/accelerate/v0.25.0/en/concept_guides/big_model_inference" target="_blank" rel="noopener nofollow ugc">Handling big models for inference</a></h3>

  <p>We’re on a journey to advance and democratize artificial intelligence through open source and open science.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Example</p>
<pre><code class="lang-auto">device_map = {
    "transformer.wte": "cpu",
    "transformer.wpe": 0,
    "transformer.drop": "cpu",
    "transformer.h.0": "disk"
}
</code></pre>
<p>Will try from my end as well</p>

chrathans: <p>Hey <a class="mention" href="/u/anantguptadbl">@anantguptadbl</a>, I am not sure how this will help me. The issue I guess is that the model is bigger than the memory capacity of my first GPU and by default, it seems that I am only using one GPU.</p>

Soumya_Kundu: <p>Haven’t explored it myself but maybe relevant: <a href="https://huggingface.co/docs/accelerate/usage_guides/big_modeling" class="inline-onebox" rel="noopener nofollow ugc">Big Model Inference</a></p>

chrathans: <p>Ok that indeed worked for me!</p>
DTensor sharding strategy support for Autograd override linear op hits issue when bias = None

satheeshhab: <p>On HPU device we have an autograd override (C++) implementation of aten::linear op. To support DTensors, I implemented op_strategy (register_op_strategy) for linear (fwd and bwd). My implementation of op strategy hits an ValueError when bias = None.</p>
<p>ValueError on TensorMeta=None in _wrap_output_spec_tensor_meta() inside torch/distributed/tensor/_sharding_prop.py</p>
<p>for i, spec in enumerate(output_specs):<br>
if isinstance(spec, DTensorSpec):<br>
output_tensor_meta_i = output_tensor_meta[i]<br>
<strong>if not isinstance(output_tensor_meta_i, TensorMeta):</strong><br>
<strong>raise ValueError(</strong><br>
<strong>f"ShardingPropagator error: output {i} does not have an associated TensorMeta"</strong><br>
<strong>)</strong><br>
spec.tensor_meta = output_tensor_meta_i</p>
<p>My question - does this ValueError check hold when a case like optional bias tensor is not supplied for linear (bias=None). Note that I am hitting this case only with autograd override implementation for linear. Without autograd override aten::linear gets decomposed into multiple other ops and we won’t hit this ValueError.</p>
Exclude specific layers from quantization

jpiat: <p>Hi,</p>
<p>i’m working on network quantization using the pytorch 2 export and AO and i would like to apply layer specific quantization strategies to my model to maximize performance. In AO when writing your quantizer, you can define how to annotate (or not to annotate) a  module/function based on its type but i don’ see any way to change quantization strategy based on the position of the module/function in the network. Let say i have a U-NET architecture and i want to avoid quantizing the first layer of the encoder, how i can achieve this ?</p>
<p>thanks</p>

jerryzh168: <p>you can probably write code to achieve this, something like:</p>
<pre><code class="lang-auto">def annotate(self, gm: GraphModule):
    linear_count = 0
    for n in gm.graph.nodes:
        if n.op == "call_module" and type(..module type) == torch.nn.Linear and linear_counter == 1:
            annotate_linear(...)
</code></pre>
Run two models in parallel using torch.multiprocessing on ONE GPU is slower than running in sequential

Hi_u_Bui_Nguy_n_Trun: <p>Hi, i’m trying to run two small models on the same GPU in parallel. But the result is not as my expect. They run slower than the sequential version.<br>
I’m trying to adjust the batchsize, the num_iter, but the parallel one always be slower. Am i missing somethings ?<br>
The code:</p>
<pre><code class="lang-auto">from torchvision.models import  resnet18, ResNet18_Weights, resnet34,ResNet34_Weights
import torch
import time
import torch.multiprocessing as mp


def model18():
    return resnet18(weights = ResNet18_Weights.IMAGENET1K_V1).cuda().eval()
def model34():
    return resnet34(weights=ResNet34_Weights.IMAGENET1K_V1).cuda().eval()
def infer(model):
    data = torch.zeros(32,3,224,224).cuda()
    output = []
    for i in range(1000):
        output = model(data).cpu().detach().numpy()
        if i%100==0:
            print(i,end=" ")
    print(f"Done {output.shape}")

if __name__=="__main__":
    mp.set_start_method("spawn")
    # m = model()
    m1 = model18()
    m2 = model34()
    time.sleep(1)
    # print(m1==m2, m1==m)
    #
    p1 = mp.Process(target=infer, args=(m1,))
    p2 = mp.Process(target=infer, args=(m2,))
    print("Inited")

    print("sequential")
    start = time.time()
    infer(m1)
    print(time.time()-start)
    infer(m2)
    print(time.time()-start)
    print("Done sequential.....................")
    print()
    print("Parallel")
    startt = time.time()
    p1.start()
    p2.start()
    print("Started:", time.time()-startt)
    p1.join()
    p2.join()
    print(time.time()-startt)
    print("Done parallel.....................")
</code></pre>
<p>The result in the console:</p>
<pre><code class="lang-auto">Inited
sequential
0 100 200 300 400 500 600 700 800 900 Done (32, 1000)
51.6720654964447
0 100 200 300 400 500 600 700 800 900 Done (32, 1000)
156.92688918113708
Done sequential.....................
Parallel
0.5700843334197998
0 0 100 100 200 300 200 400 500 300 600 700 400 800 500 900 Done (32, 1000)
600 700 800 900 Done (32, 1000)
185.4344515800476
Done parallel.....................```</code></pre>
Pickling entire dataloader when n_workers > 0 causing GCP issues

JLenz: <p>I am trying to run a distributed training run using DDP, and PyTorch lightning, alongside the MosaicML streaming library. The goal is to directly stream shards of data from a GCP bucket into my dataloader(s). Right now the process works perfectly when num_workers = 0, but when I set the value higher, I get this error message from GCP:</p>
<pre><code class="lang-auto">line 387, in _get_iterator
    return _MultiProcessingDataLoaderIter(self)
line 1040, in __init__
    w.start()
line 121, in start
    self._popen = self._Popen(self)
line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
 line 284, in _Popen
    return Popen(process_obj)
  File , line 32, in __init__
    super().__init__(process_obj)
  File 
line 19, in __init__
    self._launch(process_obj)
line 47, in _launch
    reduction.dump(process_obj, fp)
 line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
, line 194, in __getstate__
    raise PicklingError(
_pickle.PicklingError: Pickling client objects is explicitly not supported.
</code></pre>
<p>From what I understand, such as from <a href="https://discuss.pytorch.org/t/torch-distributed-and-num-workers-0-pickle-error/190357/5" class="inline-onebox">Torch distributed and num_workers&gt;0 pickle error - #5 by farakiko</a> there is an ‘issue’ where PyTorch tries to pickle each dataloader individually, which is incompatible with GCP buckets. Is there a known workaround for this issue?</p>
Does torch.compile guarantee that the generated fusion strategy and Triton kernel are the same every time?

toddlt: <ul>
<li>Will torch.compile generate different computation graphs under different runtime environments?</li>
<li>Can the cache in /tmp/torchinductor_{user} affect the generated results?</li>
</ul>

ptrblck: <p>I don’t think so as different code and optimizations will be generated for different hardware architectures.</p>
Feeding dictionary of tensors to model on gpu

Jgrietens: <p>Hi all,</p>
<p>I am using a DataLoader to feed x and y to my model during training. However, the y object is a built-in python dictionary containing 2 types of labels y = {‘target1’ : Tensor1 , ‘target2’: Tensor2}.</p>
<p>I want to load y into the gpu. However, this is not possible directly on the dict. I know that I could extract target 1 and 2, load them separately into cuda and provide this data to the model as such:</p>
<p>target1 = y[‘target1’].to(device)<br>
target2 = y[‘target2’].to(device)<br>
model(x, target1, target2)</p>
<p>But, for many reasons, I have made the design decision to feed the dict to the model directly:</p>
<p>Model(x,y)</p>
<p>Is there an elegant solution for this ? I see 2 options and for both I have no clue if they are sound:</p>
<ol>
<li>
<p>untangle Y within the .forward() method of my model. Meaning that I would extract the targets from the Y dictionary inside the .forward method and send them to the gpu inside.</p>
</li>
<li>
<p>Sending the values in my Y dictionary to the gpu separately but keep them in the dict structure.</p>
</li>
</ol>
<p>y[‘target1’] = y[‘target1’].to(device)<br>
y[‘target2’] = y[‘target2’].to(device)<br>
model(x, y)</p>
<p>For both methods I am worried about unwanted inneficiency. Could anyone help me with understanding if those methods would make sense and if not, why not and what else can I try ?</p>
<p>Thanks !</p>

ptrblck: <p>Both approaches should work and even <code>nn.DataParallel</code> should create chunks of your <code>dict</code> in case you want to unwrap it inside the <code>forward</code> method.<br>
I don’t know of any advantages of one approach over the other, if you need to use dicts.</p>

Maria_Djeblahi: <p>Hi, J<br>
I’m in the same situation I build the model to use the dictionary (the form is (src, dst): embeddings) for each edge in a graph, since the graphs are very large, I would ask you which one of the method are more suitable to load the dictionaries to the device, I would use the complete form of the dictionary inside the model?</p>
TorchScript Model Performance and Multi-Threading Issue

Ella_Mendelson: <p>Hello,</p>
<p>I’m running a TorchScript model that takes a 3-channel 640x640 float image as input, normalized to the range 0.0−1.00.0 - 1.00.0−1.0. However, my input is a single-channel <strong>uint8</strong> image.</p>
<p>Initially, I used the following code to preprocess the image:</p>
<pre><code>	cv::Mat&amp; image = images[i];
	image.convertTo(image, CV_32F, 1.0 / 255.0);  
	torch::Tensor tensor_image = torch::from_blob(image.data, { height, width, channels }, torch::kFloat).to(device);
	tensor_image = tensor_image.permute({ 2, 0, 1 });  
	tensor_images.push_back(tensor_image.clone());  
</code></pre>
<p>This implementation works correctly, but it’s slow. To improve performance, I modified it to first send the original image to the GPU and then process it:</p>
<pre><code>	cv::Mat single_channel;
	cv::extractChannel(image, single_channel, 0);
	torch::Tensor tensor_image = torch::from_blob(single_channel.data, { height, width }, torch::kByte);
	tensor_image = tensor_image.to(device, torch::kFloat).div_(255.0);
	tensor_image = tensor_image.unsqueeze(0).expand({ 3, height, width });
	tensor_images.push_back(tensor_image);
</code></pre>
<p>This approach is significantly faster. However, when running it in my multi-threaded system, it crashes.</p>
<p>Do you have any insights or suggestions on how to resolve this issue?</p>
<p>I appreciate your help in advance!</p>