akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
RuntimeError: No CUDA GPUs are available

torchlearner: <p>Hello,<br>
I am using a recent project called vLLM for running llm inference on an A100 GPU.<br>
For certain cases (like bfloat16 precision) when it tries to run torch.cuda.get_device_capability() (see code link <a href="https://github.com/vllm-project/vllm/blob/main/vllm/config.py#L345" rel="noopener nofollow ugc">https://github.com/vllm-project/vllm/blob/main/vllm/config.py#L345</a>  ), to check the device capability, I get the error below .</p>
<p>prop = get_device_properties(device)<br>
File “site-packages/torch/cuda/<strong>init</strong>.py”, line 395, in<br>
_lazy_init()  # will define _get_device_properties<br>
File “site-packages/torch/cuda/<strong>init</strong>.py”, line 247, in _lazy_init<br>
torch._C._cuda_init()<br>
<strong>RuntimeError: No CUDA GPUs are available</strong></p>
<p>Now the interesting part - When I run vLLM standalone I don’t get this error. With the same configs when I run it in my application (python server), I get this error.</p>
<p>Any. thoughts on what specific scenario or env vars might trigger this ? I have checked most standard stuff.</p>

ptrblck: <p>What does “standalone” and “Python server” mean in this context? Are you using different virtual environments? If so, make sure both have a proper PyTorch binary with CUDA dependencies installed.</p>

torchlearner: <p>Yea, by standalone I mean, when I run vllm on its own, it brings up a gunicorn server, that we can hit for inference. Where as python server is a separate server, and we integrate vllm as part of that. Now both of  these setup run with the same PYTHONPATH and environment variables.</p>
<p>Can you let me know any specific  reason why this might show up in the case I mentioned ? I know there’s no actual problem with cuda setup as it works in the direct vllm path. So I just need to understand what triggers this in the failure path.</p>

anothertorchlearner: <p>I am running into a similar issue. Is there a solution?</p>
Support loading and executing a ExportedProgram from torch.export with forward + backward + optimizer step in C++ environment

chzhu: <p>Hi all, we are currently working on an online ML platform in the company which require us to</p>
<ol>
<li>export a pytorch model graph and variable into an IR which can be executed in c++ environment</li>
<li>The graph can be executed with both forward pass, backward pass and optimizer to update the variables</li>
<li>We need to get the updated variables from training process, and send to another inference service to execute the forward pass graph with updated variables</li>
</ol>
<p>I did some doc and code search and <a href="https://pytorch.org/docs/stable/export.html" rel="noopener nofollow ugc">torch.export</a> seems to be the closest way to achieve this, but there are some gaps, not sure if I missed anything</p>
<ol>
<li>torch.export can only export forward pass and cannot export forward + backward + optimizer step all into the same graph . The backward graph is executed eagerly after loaded back in python environment (checked the latest pytorch 2.5 doc <a href="https://pytorch.org/docs/stable/export.html" rel="noopener nofollow ugc">here</a>). Also the AOT autograd seems to be exporting</li>
<li>In order to run the graph in C++, We can only compile the graph into aot_inductor and put that into .so file, there is not C++ API to load the exported graph and programically call this graph</li>
<li>There is no way to call this compute graph while passing variable update to the compute graph</li>
</ol>
<p>Do we have any plans to extend torch export to support such functionalities?</p>
Pytorch and cuda compute

prabu-github: <p>Can PyTorch 2.2 and higher be installed for a Titan X (Pascal) GPU? NVIDIA’s website says this GPU is compute capability 5.2.</p>

ptrblck: <p>Yes, all of our binaries support Pascal GPUs.</p>

prabu-github: <p>Thank you for the information <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
Tracing a model with control flow in forward operation

sim: <p>I am tracing a model to export it to c++ via the command:</p>
<p>torch.jit.trace(model.eval(), example_inp).</p>
<p>Does the tracing done as above still work if I include in the forward pass of the model something like the following code snippet?</p>
<p>def forward(self,x):<br>
if self.eval:<br>
return x + 5<br>
else:<br>
return x</p>
<p>Or I have to go through annotation to export the model even with this minimal control flow?</p>

ptrblck: <p><code>torch.jit.trace</code> will not record any control-flow as described in the <a href="https://pytorch.org/docs/stable/generated/torch.jit.trace.html">docs</a>.<br>
Your condition in your posted code is wrong since you are comparing against a function:</p>
<pre data-code-wrap="python"><code class="lang-python">model.eval
&lt;bound method Module.eval of MyModel()&gt;
</code></pre>
<p>which will always return true.<br>
Use <code>self.training</code> instead as this attribute will be changed according to <code>model.train()/.eval()</code>.<br>
After this fix, the traced model will stick to the recorded code path.</p>

sim: <p>Thanks a lot for the correction, I am not sure I got the last part of your response though. Using the following code snippet is the recorded code path going to be  the one in the else part of the forward?</p>
<p>Def forward(self,x):<br>
If self.training:<br>
Return x<br>
Else:<br>
Return x+ 5</p>
<p>Torch.jit.trace(model.eval(), example_inp)</p>

ptrblck: <p>The traced model would take the <code>else</code> path even if you want to switch back to training mode:</p>
<pre data-code-wrap="python"><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
    
    def forward(self,x):
        if self.training:
            return x
        else:
            return x + 5

x = torch.ones(10)
model = MyModel()

out = model(x)
print(out)
# tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

model.eval()
out = model(x)
print(out)
# tensor([6., 6., 6., 6., 6., 6., 6., 6., 6., 6.])

# reset
model.train()
out = model(x)
print(out)
# tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

model_traced = torch.jit.trace(model.eval(), x)

print(model_traced(x))
# tensor([6., 6., 6., 6., 6., 6., 6., 6., 6., 6.])

# does not change the behavior
model_traced.train()
print(model_traced(x))
# tensor([6., 6., 6., 6., 6., 6., 6., 6., 6., 6.])
</code></pre>

sim: <p>ok now I understand, thank you a lot!</p>
Trying to setup a service, need some quick help with pytorch/cuda compatibility

dillonius: <p>Trying to setup a python service. Hosting it on AWS with a g4dn.xlarge GPU using Amazon Linux 2023.</p>
<p>Driver: CUDA 12.7<br>
Toolkit: CUDA Toolkit 12.6 Update 2<br>
Pytorch: 2.5.1+cu124</p>
<p>I’ve been spinning on this trying different things. Do I just need to downgrade my toolkit or pytorch version or something obvious? Wanted to seek some clarification before I keep going round and round.</p>
<p>±----------------------------------------------------------------------------------------+<br>
| NVIDIA-SMI 565.57.01              Driver Version: 565.57.01      CUDA Version: 12.7     |<br>
|-----------------------------------------±-----------------------±---------------------+<br>
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |<br>
|                                         |                        |               MIG M. |<br>
|=========================================+========================+======================|<br>
|   0  Tesla T4                       Off |   00000000:00:1E.0 Off |                    0 |<br>
| N/A   31C    P0             26W /   70W |       1MiB /  15360MiB |     10%      Default |<br>
|                                         |                        |                  N/A |<br>
±----------------------------------------±-----------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------------------+<br>
| Processes:                                                                              |<br>
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |<br>
|        ID   ID                                                               Usage      |<br>
|=========================================================================================|<br>
|  No running processes found                                                             |<br>
±----------------------------------------------------------------------------------------+</p>
<p>Thanks for any help.</p>

ptrblck: <p>It’s unclear what the issue is. Your setup looks valid and the locally installed CUDA toolkit won’t be used unless you build PyTorch from source or a custom CUDA extension.</p>

dillonius: <p>The issue was that it wasn’t finding a CUDA gpu. I ended up changing the instance to Ubuntu and it fired right up. I don’t really know what happened. But it works, so whatever, haha.</p>

dillonius: <p>Just kidding. It was working because of a backup I built in, and it’s only using the CPU. So yeah, back to where I was at. Sigh.</p>

dillonius: <p>I was finally able to get it working properly btw. Downgraded the drivers to 12.1 and that did the trick.</p>
Are Random Functions Deterministic Given Seed?

zhixuan-lin: <p>Is there any guarantee that when given the same seed, PyTorch random functions (e.g., <code>torch.randperm</code> ) will always generate the same results in any case (e.g., across different PyTorch versions, machines, GPU or CPU)?</p>

Craig_Hicks1: <p>No, PyTorch CPU and GPU use different random number generators (RNGs), which means that even if you set the same seed, you will get different random numbers on the CPU and GPU.</p>
<p>Here’s why:</p>
<ul>
<li><strong>Different Algorithms:</strong></li>
</ul>
<p>CPU and GPU use different algorithms for random number generation, which leads to different sequences even with the same seed.</p>
<ul>
<li><strong>Hardware Differences:</strong></li>
</ul>
<p>GPUs are optimized for parallel processing, which can affect the way they generate random numbers compared to CPUs.</p>
<p>How to Ensure Reproducibility:</p>
<ul>
<li><strong>Use the Same Device:</strong></li>
</ul>
<p>If you need reproducible results, always use the same device (either CPU or a specific GPU) for your experiments.</p>
<ul>
<li><strong>Set the Seed:</strong></li>
</ul>
<p>Use <code>torch.manual_seed()</code> to set the seed for the CPU RNG and <code>torch.cuda.manual_seed()</code> for the GPU RNG.</p>
<ul>
<li><strong>Save and Load RNG State:</strong></li>
</ul>
<p>If you need to reproduce results across different sessions, you can save the RNG state using <code>torch.get_rng_state()</code> and load it later using <code>torch.set_rng_state()</code>.</p>
<p>As for different PyTorch versions, that depends on whether the algorithm used, and its implementation, is exactly the same, or not.</p>
<p>As for <code>torch.randperm</code>, the <a href="https://github.com/pytorch/pytorch/blob/c4ae45104f87e96dcd6d5ba47cb4e356fc2c63f3/aten/src/ATen/native/cuda/Randperm.cu#L23" rel="noopener nofollow ugc">in-source documentation</a> shows that internally it calls <code>argsort</code> (or maybe <code>sort</code> ?) with <code>stable=True</code>.  That implies the result will be always be the same if the RNG+seedstate is the same.</p>
<p>I have an unanswered question about whether <code>argsort</code>/<code>sort</code> with <code>stable=False</code> is always reproducible (a.k.a. “deterministic” given the same inputs).  It could conceivably be reproducible, but that depends on whether there are potential race conditions in the algorithm.  That is a separate issue from the RNG+seed.</p>
<p>See also <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">Randomness</a></p>
The `.forward()`/etc. shape API

kuraga: <p><a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.forward" rel="noopener nofollow ugc">There is</a> the <code>nn.Module.forward(...)</code> method. So,  it’s “standardized”.</p>
<p>Each concrete <code>.forward(...)</code>'s documentation has its return shape (formula).</p>
<p>Is there an API/a project of API of the <code>.get_forward_shape(...)</code> method to get it programatically?</p>

kuraga: <p>/cc <a class="mention" href="/u/alband">@albanD</a>, <a class="mention" href="/u/ptrblck">@ptrblck</a> , thanks.</p>

kuraga: <p>I’ve reposted at <a href="https://github.com/pytorch/pytorch/issues/140703" rel="noopener nofollow ugc">the issues</a>.</p>
Seeking advice on computing backprop step by chunks

meditans: <p>Hi,<br>
I am facing a challenge with autograd when a computation <em>temporarily</em> involves a very large tensor that won’t fit on my GPU. If there wasn’t a need to retain gradients, a chunk-by-chunk calculation would work. However, autograd requires that I store this intermediate large tensor on my GPU.<br>
Consider a pipeline: small tensor <code>a</code> → very large tensor <code>b</code> → small tensor <code>c</code>. It seems feasible that if the forward pass can be calculated chunk-by-chunk, so could the gradient of <code>c</code> with respect to <code>a</code>.</p>
<ul>
<li>Concrete Scenario:<br>
Two tensors <code>x</code>, <code>y</code> of shape <code>(256, 256, 128)</code> and <code>(256, 1024, 128)</code> combine to form a large tensor <code>L</code> of shape <code>(256, 256 * 1024, 2 * 128)</code>, by concatenating channels for each vector pairing from <code>x</code> and <code>y</code>. A series of linear layers then reduce this to a smaller tensor <code>z</code> of shape <code>(256, 256 * 1024, 1)</code>.</li>
</ul>
<p>I would like to backpropagate (the gradients of) <code>z</code> with respect to <code>x</code> and <code>y</code>. The way it works in my mind is: <code>x</code> and <code>y</code> are stored, the forward computation computes <code>z</code> by chunks, so it never has to materialize <code>L</code> completely. The backward computation calculates the gradients by chunks (this is possible because the linear layers are well behaved), similarly never materializing <code>L</code> completely.</p>
<ul>
<li>Potential Solutions Explored:
<ul>
<li>A custom autograd function and class like the <a href="https://pytorch.org/docs/stable/notes/extending.html#example" rel="noopener nofollow ugc">example in the docs</a>. This almost suit my needs, but I would have to manually pass in tensors (weights and biases) for the linear layers (a lot of tensors if I have multiple linear layer, and I would have to reconstruct all the functionality of <code>nn.Linear</code> myself).</li>
<li>Gradient checkpointing, although in the backprop pass I still materialize the whole <code>L</code> tensor.</li>
</ul>
</li>
</ul>
<p>Any suggestions on handling this scenario would be appreciated!</p>

meditans: <p>Figured it out. I could directly pass a <code>nn.Module</code> in my custom autograd <code>Function</code>, and that was an acceptable design.</p>
Connect [127.0.1.1]:20892: Connection refused

mingyu: <p>Hi! I have two jetson devices, A and B, both of which are connected to the same switch, with one IP address of 192.168.1.101 and the other 192.168.1.153. I want to use the RPC protocol to make two devices do a simple task (device A passes an int variable to device B, and device B does a multiplication of this variable and outputs it). Both devices have 1.11.0 version of torch, and below is the code for both of my devices.</p>
<p>A Equipment:<br>
import torch<br>
import torch.distributed.rpc as rpc<br>
import torch.distributed as dist</p>
<p>def send_data_to_device_b():<br>
a = 3<br>
print(f"Device A sending data: a = {a}")</p>
<pre><code>rpc.rpc_sync("device_b", add_and_print, args=(a,))
</code></pre>
<p>def init_rpc():<br>
print(“Initializing RPC on device A…”)<br>
dist.init_process_group(backend=“gloo”, init_method=“tcp://192.168.1.101:20892”, world_size=2, rank=0)</p>
<pre><code>rpc.init_rpc("device_a", rank=0, world_size=2)

print("Sending data to device B...")
send_data_to_device_b()

print("Shutting down RPC...")
rpc.shutdown()
</code></pre>
<p>if <strong>name</strong> == “<strong>main</strong>”:<br>
init_rpc()</p>
<p>B equipment:<br>
import torch<br>
import torch.distributed.rpc as rpc<br>
import torch.distributed as dist</p>
<p>def add_and_print(a):<br>
b = 4<br>
print(f"Device B received: a = {a}, b = {b}. The result of a + b is: {a + b}")</p>
<p>def init_rpc():<br>
dist.init_process_group(backend=“gloo”, init_method=“tcp://192.168.1.101:29520”, world_size=2, rank=1)</p>
<pre><code>rpc.init_rpc("device_b", rank=1, world_size=2)

rpc.shutdown()
</code></pre>
<p>if <strong>name</strong> == “<strong>main</strong>”:<br>
init_rpc()</p>
<p>I started the code on device B first, and then started the code on device A, and device A returned the following error message.<br>
Initializing RPC on device A…</p>
<p>Traceback (most recent call last):<br>
File “rpc_a.py”, line 30, in <br>
init_rpc()<br>
File “rpc_a.py”, line 16, in init_rpc<br>
dist.init_process_group(backend=“gloo”, init_method=“tcp://192.168.1.101:29520”, world_size=2, rank=0)<br>
File “/home/nvidia/anaconda3/envs/py3.8/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 602, in init_process_group<br>
default_pg = _new_process_group_helper(<br>
File “/home/nvidia/anaconda3/envs/py3.8/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 703, in _new_process_group_helper<br>
pg = ProcessGroupGloo(prefix_store, rank, world_size, timeout=timeout)<br>
RuntimeError: [/media/nvidia/NVME/pytorch/JetPack_5.0/pytorch-v1.11.0/third_party/gloo/gloo/transport/tcp/pair.cc:799] connect [127.0.1.1]:20892: Connection refused</p>
<p>And device B has been waiting for the connection and nothing has returned. I confirmed that the firewalls of both devices are off and can ping each other at the same time.<br>
What is the problem and how to fix it?<br>
Any suggestion would be very useful!</p>
Busy Waiting During Synchronization

NoSo: <p>Hello, I have a pretty odd request.</p>
<p>I have a pretty simple program that uses huggingface and nn.DataParallel to finetune a model using 4 GPUs. (Please correct me if I’m wrong) During the forward, backward, and the optimizer update steps, I would like to add busy waiting while GPUs wait on synchronization. Essentially the goal is to maintain each GPU’s utilization so it stays at 100%. After doing some investigation (as outlined in <a href="https://www.microsoft.com/en-us/research/uploads/prod/2024/03/GPU_Power_ASPLOS_24.pdf" rel="noopener nofollow ugc">Microsoft’s paper</a>), I see that util and power both take a dip during synchronization phases. I know this seems a bit odd, but it’s for a larger project that I am working on.</p>
<p>So how can I launch a kernel to do some busy waiting on the GPUs during synchronization?</p>
Row deletion of sparse tensors

ParticularCrumble: <p>Hello all! I am currently working with Kronecker products which lead to incredible amounts of sparsity and dense tensors are not feasible currently as they spill from my vram. Moreover, I am not working with neural nets, I am trying to do kernel matrix computation for SVM’s so doing things in batches is not the most straightforward. Thus I am turning to sparse tensors. What I would like to do is given some sparse tensor, delete rows based on a mask vector: v = [1 0 1 0 0 0], where if there is a 1 in ith entry in v, then the ith row of my sparse tensor is made into 0’s. I tried just doing sparse_tensor[v], but this gave an error saying that this doesn’t work on sparse tensors. Is there any work around without looping through the rows?</p>

ParticularCrumble: <p>Was able to fix this by using the sparse kronecker code here: <a href="https://github.com/pytorch/pytorch/issues/134069" class="inline-onebox" rel="noopener nofollow ugc">Kron with gradient for sparse tensors · Issue #134069 · pytorch/pytorch · GitHub</a></p>
<p>For deleting rows I did this:<br>
allowed_indices = your_allowed_indices</p>
<p>keep_mask = torch.isin(new_indices[0], allowed_indices)<br>
new_values = new_values[keep_mask]</p>
<p>keep_mask = keep_mask.expand(2,new_indices.size()[1])<br>
new_indices = new_indices[keep_mask].view(2, -1)</p>
<p>This deleted both the indices and the values of the resulting sparse kronecker product.</p>
Cuda support for MAC available?

EarlyBird: <p>There is a Cuda driver at NVIDIA available for MAC.<br>
Is there any support for pytorch on MAC for Cuda?</p>
<p>My MAC:<br>
Power Book 15"<br>
OSX 10.15.7</p>

ptrblck: <p>No, since macOS support was dropped in CUDA 11.0.</p>

EarlyBird: <p>Thank you! <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"> Very precise.</p>

alanngnet: <p>No, but that is an irrelevant question since Apple didn’t use CUDA-relevant GPU hardware in its Intel era (at least in my experience), and then with the M-family hardware we now use PyTorch with MPS libraries instead of CUDA to get full hardware acceleration on Macs.</p>
Multi-GPU training with training loop that can skip backpropagation

RomStriker: <p>Hi,<br>
I am training a model which for certain batches has no loss (and skip backprop) when certain conditions are met regarding the model output. I am trying to train this model using DDP, however, if one of the GPUs has no loss, it will not perform backprop, and the other GPUs will wait indefinitely for the gradients of this GPU, leading to a timeout. Is there a way to deal with such cases? The code works fine for a single GPU. I tried to create a dummy loss and use that to perform backprop but it did not help. Any help would be appreciated.</p>
<pre><code class="lang-auto">            if self.args.amp:
                optimizer.zero_grad()
                # to check for cases when there is nothing to optimize
                if losses.grad_fn is None:
                    losses = losses + torch.zeros(1, requires_grad=True).to(losses.device)
                scaler.scale(losses).backward()
                # Check if there are valid gradients before stepping the optimizer
                valid_gradients = any(p.grad is not None for p in model.parameters() if p.requires_grad)
                if valid_gradients:
                    if self.args.clip_max_norm &gt; 0:
                        scaler.unscale_(optimizer)
                        torch.nn.utils.clip_grad_norm_(model.parameters(), self.args.clip_max_norm)
                    scaler.step(optimizer)
                    scale = scaler.get_scale()
                    scaler.update()
                    skip_lr_step = (scale &gt; scaler.get_scale())
                else:
                    print("Skipping backprop step.")
                    skip_lr_step = True
</code></pre>
How encoder-decoder transformer translate 2 sentences as an input

Mert_Oguzhan: <p>Hi everyone! Do you have any idea how encoder-decoder transformer translate 2 sentences as an input  ? For instance; I am passing 2 sentences as an input through encoder. “This is a dog. Is that correct answer?” After decoder predict “dog” and “.” then should predict &lt;(eos)&gt; (end of sequence). How encoder-decoder transformer translate (separate) 2 sentences as an input like this? Furthermore, Does encoder-decoder transformer use segment embedding for this situation?</p>
Torchrun assigns same LOCAL_RANK to processes sharing node

chazm: <p>Hello, I am attempting to do DDP on a SLURM cluster. More specifically, each node is 2 GPU, and I have 4 nodes I can assign to my job. Because of this, I am submitting an sbatch to run 8 tasks, with 2 tasks per node:</p>
<pre><code class="lang-auto">#!/bin/bash

#SBATCH --job-name=csinet-rewrite
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=4
#SBATCH --partition=gpu
#SBATCH --gres=gpu:2
#SBATCH --constraint=gpu_32gb

export OMP_NUM_THREADS=4
export NCCL_DEBUG=INFO
export MASTER_ADDR=$(scontrol show hostname ${SLURM_JOB_NODELIST} | head -n 1)

module load anaconda
conda activate torch

#Checking ports on Master
netstat -lnt

srun nvidia-smi -L
srun torchrun \
	--nnodes 4 \
	--nproc_per_node 2 \
	--rdzv_id $RANDOM \
	--rdzv_backend c10d \
	--rdzv_endpoint $MASTER_ADDR:29400 \
	main.py
</code></pre>
<p>While running this, my model fails with a NCCL error stating that separate ranks are attempting to access the same GPU. This should not be possible, as I am running torch.cuda.set_device(environ[“LOCAL_RANK”]) first thing:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://pastebin.com/dhT3H8R3">
  <header class="source">
      <img src="https://pastebin.com/favicon.ico" class="site-icon" width="16" height="16">

      <a href="https://pastebin.com/dhT3H8R3" target="_blank" rel="noopener nofollow ugc">Pastebin</a>
  </header>

  <article class="onebox-body">
    <img width="" height="" src="https://pastebin.com/i/facebook.png" class="thumbnail">

<h3><a href="https://pastebin.com/dhT3H8R3" target="_blank" rel="noopener nofollow ugc">Active Internet connections (only servers)Proto Recv-Q Send-Q Local Address -...</a></h3>

  <p>Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Looking at the debug info, it appears that torchrun is assigning LOCAL_RANK values without regard to which node each task is on. How do I fix this?</p>

tj.solergibert: <p>Hi, try setting <code>#SBATCH --ntasks-per-node=1</code> instead of 2!</p>
Inserting Unnecessary Fake Quants during QAT?

jsenn: <p>I’m confused about the way PyTorch inserts FakeQuant nodes during Quantization-Aware Training. In the <a href="https://pytorch.org/blog/quantization-aware-training/" rel="noopener nofollow ugc">documentation</a>, the following image appears:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/1/4119adbe4a32a307a9900af174eb1f1d3ecb9957.png" data-download-href="https://discuss.pytorch.org/uploads/default/4119adbe4a32a307a9900af174eb1f1d3ecb9957" title="fg2"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/4119adbe4a32a307a9900af174eb1f1d3ecb9957_2_690x191.png" alt="fg2" data-base62-sha1="9hU3dx44qLTtrkoF905IvbjaOq3" width="690" height="191" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/4119adbe4a32a307a9900af174eb1f1d3ecb9957_2_690x191.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/4119adbe4a32a307a9900af174eb1f1d3ecb9957_2_1035x286.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/1/4119adbe4a32a307a9900af174eb1f1d3ecb9957_2_1380x382.png 2x" data-dominant-color="EBEBEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">fg2</span><span class="informations">3488×966 232 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I understand the FakeQuant after W, since it will actually be quantized. But why insert the FakeQuant node after the input x, which necessitates the Quant/Dequant overhead at inference time? The Linear node is in fp32 anyway, so it seems this is pure overhead.</p>
<p>I’ve noticed this in practice: models exported to ONNX from PyTorch have zillions of Quant/Dequant pairs interspersed throughout, which seem unnecessary.</p>

HDCharles: <p>A) this is for dynamic quantization where you quantize the activations<br>
B) the documentation you cite is for a lowering process, i.e. you do all the math needed to figure out what the quantization parameters and weights would be (prepare), then you get the graph to a special format (convert) which is not necessarily performant, then you use a lowering technique to recognize these special forms and swap them for performant ones. This lowering step would recognize the above graph as a dynamically quantized llinear op, where the actual graph is</p>
<p>x_fp →  lowered_q_linear → out_fp<br>
w_q--------^</p>

jsenn: <p>Thank you, this is exactly what I was missing. So this is a sort of intermediate representation that’s expected to be optimized down into quantized kernel.</p>
How to minimize the gaps between kernels on GPU?

Zhewen_Hu: <p>I’m currently training multiple small models across several CUDA streams to achieve parallelism with Libtorch. However, each kernel execution is very short (lasting only a few microseconds) and scattered across the timeline, which means they rarely overlap or parallelize effectively. I’ve attached a screenshot showing how dispersed the kernel executions are.</p>
<p>I’m wondering if it is due to the overhead of launching each kernel? And if so, is there any way to minimize the gaps between these kernels to improve parallelism? Any suggestions would be appreciated!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/3/331fdd7825798204517c3db8bce5c9c5d7a8b7a2.png" data-download-href="https://discuss.pytorch.org/uploads/default/331fdd7825798204517c3db8bce5c9c5d7a8b7a2" title="Snipaste_2024-11-11_22-54-35"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/331fdd7825798204517c3db8bce5c9c5d7a8b7a2_2_690x121.png" alt="Snipaste_2024-11-11_22-54-35" data-base62-sha1="7igCxLajXvGV2quwjmr1qkMLduy" width="690" height="121" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/331fdd7825798204517c3db8bce5c9c5d7a8b7a2_2_690x121.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/331fdd7825798204517c3db8bce5c9c5d7a8b7a2_2_1035x181.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/3/331fdd7825798204517c3db8bce5c9c5d7a8b7a2_2_1380x242.png 2x" data-dominant-color="F4F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Snipaste_2024-11-11_22-54-35</span><span class="informations">1878×330 45.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>If your workload suffers from slow kernel launches, you could check if you could use CUDAGraphs for your use case either <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-graphs">directly</a> or via <code>torch.compile</code>.</p>

Zhewen_Hu: <p>Thanks for the help. I am using Libtorch, does it also have torch.compile? I couldn’t find the document of it for Libtorch</p>

ptrblck: <p>No, I don’t think <code>torch.compile</code> or any of its stack is exposed as a pure C++ API. You could try to <code>torch.export</code> your model as described <a href="https://pytorch.org/docs/stable/export.html">here</a>.</p>
Where can I get all the `operators` of torch?

shaoyu_young: <p>Hi, guys.<br>
I want to know how to collect all the <code>operators</code> of torch (<code>torch.nn.Conv2d</code>, <code>torch.clamp</code>, etc.)</p>
<p>NOT include other APIs like <code>torch.utils.model_zoo.load_url</code></p>
<p>Thx in advance</p>

shaoyu_young: <p>I mean where can I get all these ops below<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f2ca0ae45d7a6103d15f9e3a73e3742bac831e8f.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f2ca0ae45d7a6103d15f9e3a73e3742bac831e8f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2ca0ae45d7a6103d15f9e3a73e3742bac831e8f_2_432x500.jpeg" alt="image" data-base62-sha1="yDOqacEu2MjK1GvGxQXqol4X475" width="432" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2ca0ae45d7a6103d15f9e3a73e3742bac831e8f_2_432x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2ca0ae45d7a6103d15f9e3a73e3742bac831e8f_2_648x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2ca0ae45d7a6103d15f9e3a73e3742bac831e8f_2_864x1000.jpeg 2x" data-dominant-color="E9EDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">961×1111 133 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><a class="mention" href="/u/chillee">@Chillee</a> created this figure in <a href="https://dev-discuss.pytorch.org/t/where-do-the-2000-pytorch-operators-come-from-more-than-you-wanted-to-know/373">this post</a>, but I’m unsure if the scripts to create the data are available.</p>
Handling a large dataset without loading to memory

miladiouss: <p>I have a few TB of tiny images and I would like to scan/classify them with a model I have trained. Currently, I load a bunch to memory, create a DataLoader object, run them through the model, and move to the next bunch. By using the DataLoader I get to use the same image transformation I used for the model. However, this is a little painful. Is there a better way of running a model over millions of files?</p>

ptrblck: <p>You could use a <code>Dataset</code> to lazily load all images.<br>
Have a look at <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html#dataset-class">this tutorial</a>.<br>
Basically you can pass the image paths to your <code>Dataset</code> and just load and transform the samples in <code>__getitem__</code>.</p>

miladiouss: <p>That was easy, I got it to work. I keep forgetting how flexible PyTorch is compared to other frameworks. Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>

ptrch_c_m: <p>I also have a large data set containing series, meaning that each sample of the data set is a sequence, and the data set does not fit into memory. I am looking for a way to handle this efficiently.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> if I understand your answer correctly, you suggest that each sample of the dataset will be a separate file, so that it has its own path, which will be given to the <code>DataLoader</code>. Doing that, each sample will be read each time (regarding epochs) from the disk. This would be very efficient considering RAM, but will have poor running time performance.</p>
<p>Wouldn’t be better if the dataset was split into chunks that fit into memory, and iteratively loading these chunks to memory, and then the <code>DataLoader</code> loading them from RAM?</p>
<p>This is the first time that I face RAM problem, so it would help a lot if you have any best practices regarding large datasets that don’t fit into memory.</p>
Wrong/bad inferring

igjaegle: <p>Hi, I am loading a multi-class Particle IDentification (PID) model into a C++ code and loop over all the events contains in a ROOT file to PID the tracks. The inferring is giving strange results. Even if I am using the training ROOT file, I am not getting back the expected results. How should I approach this problem?</p>
<p>tks ig.</p>
Print(torch.cuda.is_available) return false

Dong1: <p>Looks like the cuda is installed but is never activated in my Jetson, can provide some help?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e7381e98923704f3fc1980fc3d44eca827ec65df.png" data-download-href="https://discuss.pytorch.org/uploads/default/e7381e98923704f3fc1980fc3d44eca827ec65df" title="Screenshot from 2024-11-12 12-51-47"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7381e98923704f3fc1980fc3d44eca827ec65df_2_404x500.png" alt="Screenshot from 2024-11-12 12-51-47" data-base62-sha1="wZswVRn3vc8U2O5KDKh65zyOlS7" width="404" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7381e98923704f3fc1980fc3d44eca827ec65df_2_404x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7381e98923704f3fc1980fc3d44eca827ec65df_2_606x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7381e98923704f3fc1980fc3d44eca827ec65df_2_808x1000.png 2x" data-dominant-color="38162E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-11-12 12-51-47</span><span class="informations">817×1010 130 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>It seems you have installed the CPU-only PyTorch binary, so install a binary built for Jetson.</p>
`torch.multinomial`: sampling once vs in batches gives different results

marinegor: <p>Hi everyone,</p>
<p>I assumed that the following two functions would produce exactly the same results, as they’re equivalent to sampling with the concatenated weights as in <code>draw_once</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">def draw_once(seed: int):
    torch.manual_seed(seed)
    weights = torch.arange(15).reshape(3, 5).float() * 100
    return torch.multinomial(torch.vstack([weights, weights, weights]), 1).squeeze(1)


def draw_many(seed: int):
    torch.manual_seed(seed)
    weights = torch.arange(15).reshape(3, 5).float() * 100
    return torch.vstack([torch.multinomial(weights, 1) for _ in range(3)]).squeeze(1)


# draw_once(0)
# tensor([2, 0, 1, 4, 1, 4, 2, 0, 1])
# draw_many(0)
# tensor([2, 0, 1, 3, 3, 0, 4, 1, 3])


once = torch.vstack([draw_once(i) for i in range(100_00)])
many = torch.vstack([draw_many(i) for i in range(100_00)])

once.float().std(axis=0) - many.float().std(axis=0)
once.float().mean(axis=0) - many.float().mean(axis=0)
</code></pre>
<p>however, sampling results in different, i.e. means are different beyond 5 sigma everywhere but in the first drawn sample, where the sample is exactly the same every time.</p>
<p>What can be the reason for this? I’ve also tried running with a newly instantiated <code>Generator</code>, but I get the same results – first drawn sample is the same, others aren’t.</p>

ptrblck: <p>I’m unsure if I understand your question correctly, but seeding the pseudorandom number generator (PRNG) will guarantee to return the same <em>sequence</em> of random numbers for the <em>same</em> calls into the PRNG. Our two approaches use different calls into <code>multinomial</code> and are thus not guaranteed to return the same sequence of random numbers.</p>

marinegor: <p>you mean that in between the calls to <code>multinomial</code> in the loop can be some internal calls to the PRNG that I’m not calling explicitly, but they still affect the sequence?</p>
<p>It makes sense to me, though that the first <code>n</code> samples do match doesn’t really make sense.</p>
<p>Is there a way to look at the PRNG’s state, and which numbers are being generated that don’t end up in the <code>multinomial</code>?</p>

ptrblck: <p>No, I mean the calls themselves are not the same and thus not guaranteed to return the same values. E.g. the calls would advance the offset of the PRNG in different ways and the next call could thus return a different output.</p>
Attempting to run cuBLAS, but there was no current CUDA context!

janssenn: <p>Hi,</p>
<p>i want to calculate a jacobian of an linear layer using torch.func.jacrev(). Unfortunately when doing so i get the warning shown in my minimal example below. Why does this warning occur? When I use it in my main code the runtime of different code blocks changes.</p>
<pre><code class="lang-auto">import torch

dummy = torch.tensor(1.0).cuda() # dummy to init cuda
print(torch.cuda.is_initialized())

f = torch.nn.Sequential(
        torch.nn.Linear(in_features=16, out_features=16, bias=False),
    ).cuda()

X_k = torch.rand(16).cuda()
H = torch.func.jacrev(f)(X_k)

print(H.shape)
</code></pre>
<p>Output Warning:</p>
<pre><code class="lang-auto">python3.10/site-packages/torch/autograd/graph.py:769: UserWarning: Attempting to run cuBLAS, but there was no current CUDA context! Attempting to set the primary context... (Triggered internally at ../aten/src/ATen/cuda/CublasHandlePool.cpp:135.)
  return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
</code></pre>

amisu: <p><a class="mention" href="/u/janssenn">@janssenn</a> I have the same issue. Any idea how to fix this?</p>

ptrblck: <p>This warning was introduced in <a href="https://github.com/pytorch/pytorch/pull/120131">this PR</a> setting the primary context to the current one for multi-threaded use cases which could otherwise fail.</p>

amisu: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, could you clarify how the warning relates to multi-threading?</p>
<p>If I understand correctly, the warning appears even when running the “vanilla” code with a single GPU and a single thread. Do you have any suggestions on adjustments to prevent the warning from appearing?</p>

ptrblck: <p>PyTorch uses a separate thread in the backward pass. If no other CUDA operation was used before the cuBLAS call, the warning will be raised as we will set the primary context to the current one.</p>

Eurus-Way: <p>Hi, I encountered a similar problem when I was running the WGAN-GP code. I have already set the primary context to the current one for multi-threaded use cases, but the problem still exists.（There is no error when I remove the gradient penalty part in the program.）The following is the content of the error：D:\anaconda3\envs\py312\Lib\site-packages\torch\autograd\graph.py:825: UserWarning: Attempting to run cuBLAS, but there was no current CUDA context! Attempting to set the primary context… (Triggered internally at C:\cb\pytorch_1000000000000\work\aten\src\ATen\cuda\CublasHandlePool.cpp:135.)<br>
return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass   Could you give me some advice, thanks！！！</p>

ptrblck: <p>It’s not an error but an expected warning raised once if the primary context in a new thread wasn’t set already.</p>