akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
PyTorch support for NVIDIA RTX 5090

username: prashant_mittal
cooked: <p>Is PyTorch nightly build supporting NVIDIA RTX5090? My vision program was running fine till last week with nightly build  &amp; then it started giving errors.</p>
<p>I pulled stable release and it gives error below -</p>
<p>NVIDIA RTX 5090 with CUDA capability sm_120 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_70 sm_75 sm_80 sm_86 sm_90.<br>
If you want to use the NVIDIA GeForce RTX 5090 GPU with PyTorch, please check the instructions at <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">https://pytorch.org/get-started/locally/</a></p>
<p>warnings.warn(…</p>

username: ptrblck
cooked: <p>Yes, all of our nightly and stable releases using CUDA 12.8+ support Blackwell architectures.<br>
Based on the error message it seems you are using a PyTorch binary with CUDA &lt;=12.6.</p>
Fastest way to batch multi_dot

username: Ronan_Gautier
cooked: <p>I want to perform the following matrix multiplication,</p>
<pre><code class="lang-auto">(k, N, N) @ (b, N, N) @ (k, N, N) -&gt; (b, N, N)
</code></pre>
<p>which can be achieved in many different ways using the various pytorch matrix multiplication functions. But what is the fastest way to achieve that for large matrices? Below is a benchmark of three different ways (<code>tensordot</code> seems to be the fastest for now) but I can’t help but wonder: are there other faster implementations I’m missing?</p>
<pre><code class="lang-python">import torch
import timeit

k = 3
b = 4
N = 100
device = torch.device("cuda")
dtype = torch.complex64

A = torch.randn(k, N, N, dtype=dtype, device=device)
X = torch.randn(b, N, N, dtype=dtype, device=device)

# 83.8 µs ± 1.41 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
%timeit (A[:, None] @ X[None, :] @ A[:, None].adjoint()).sum(0)
# 240 µs ± 211 ns per loop (mean ± std. dev. of 7 runs, 1,000 loops each)
%timeit torch.einsum('aij,bjk,akl-&gt;bil', A, X, A.adjoint())
# 71.9 µs ± 84.7 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
%timeit torch.tensordot(torch.tensordot(A, X, dims=([2], [1])), A.adjoint(), dims=([0, 3], [0, 1]))
</code></pre>

username: Ronan_Gautier
cooked: <p>Any idea on this would be really appreciated ! It is a core operation of a really time-sensitive part of the my code, so any gain is really big.</p>
<p>I’m going with the <code>tensordot</code> approach for now which seems fastest, but it doesn’t feel efficient to run two operations for a somewhat symmetric operation such as</p>
<pre><code class="lang-python">(A[:, None] @ X[None, :] @ A[:, None].T).sum(0)
</code></pre>
<p>Thanks for your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: f3ba
cooked: <p>Hi, did you figure out a better way to do this?</p>
Python Kernel crash

username: Maciej_Mamczur111
cooked: <p>I’m using archlinux and rx 9070xt gpu. I have problem with running this: "model_1.eval()</p>
<p>with torch.inference_mode():<br>
y_preds = model_1(X_test)<br>
y_preds" because when I’m trying to do this python Kernel just crash.</p>
<p>This pice of code is from: <a href="https://github.com/mrdbourke/pytorch-deep-learning/blob/main/extras/solutions/01_pytorch_workflow_exercise_solutions.ipynb" class="inline-onebox" rel="noopener nofollow ugc">pytorch-deep-learning/extras/solutions/01_pytorch_workflow_exercise_solutions.ipynb at main · mrdbourke/pytorch-deep-learning · GitHub</a><br>
What to do?</p>
How to install pytorch when cuda=12.0?

username: 2d28xoxk
cooked: <p>I’m working on ubuntu server, got this when I want to install through pip:</p>
<pre><code class="lang-auto">RuntimeError(CUDA_MISMATCH_MESSAGE.format(cuda_str_version, torch.version.cuda))
    RuntimeError:
    The detected CUDA version (12.0) mismatches the version that was used to compile
    PyTorch (11.8). Please make sure to use the same CUDA versions.
    
    [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
</code></pre>
<p>From this my cuda =12.0 while pytorch=11.8 ,it’s becuz I didn’t find pytorch 12.0 in <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">https://pytorch.org/get-started/locally/</a> when installing.</p>
<p>So I should reinstall pytorch 12.0 now, how to do that?</p>
<p>here’s some info I can provide:</p>
<p>nvidia-smi output:</p>
<pre><code class="lang-auto">VVIDIA-SMI 525.105.17  Driver Version: 515.105.17  CUDA version 12.0
</code></pre>
<p>nvcc -V output:</p>
<pre><code class="lang-auto">Cuda comilation tools, release 12,0 V12.0.76
Build cuda_12.0.r12.0/compiler.31968024_0
</code></pre>
<p>I want to install pytorch in anaconda environment with python=3.10</p>
<p>How can I do? Thanks!!!</p>
<p>Btw I installed cuda in my personal directory, like this<br>
in ~/.bashrc (of course my own )</p>
<pre><code class="lang-auto"># cuda
export PATH="/mydirectoyr/cuda-12.0/bin:$PATH"
export :D_LBRARY_PATH="/mydirectory/cuda/cuda-12.0/lib64:/mydirectory/cuda/cuda-12.0/mylib/lib64:$LD_LIBRARY_PATH"
</code></pre>
<p>Should I choose downgrade cuda ? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=14" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>This error message is most likely raised if you try to build a custom CUDA extension in an environment where the locally installed CUDA toolkit and the CUDA toolkit used to build PyTorch differ in the major version. In this case try to install any PyTorch binary with CUDA 12 and try to rebuild your extension.</p>

username: Hamza_Javaid
cooked: <p>CUDA 12.1 build works perfectly with CUDA 12.0 due to backward compatibility as far as I know.</p>
<p>Try force-reinstall</p>
<p><code>pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu121 --force-reinstall</code></p>

username: ptrblck
cooked: <p>I wouldn’t recommend installing such an old binary since all of our CUDA 12.x builds will work on CUDA 12 drivers. As previously explained the error message seems to be raised by an attempt to build a custom extension from source.</p>
Building Torch as a Static Library Issues

username: jadenyouib
cooked: <p>I am currently trying to build Torch as a static library with the following commands in Windows in the Visual Studio Native Tools Command Prompt.</p>
<pre><code class="lang-auto">mkdir build_libtorch
cd build_libtorch
set BUILD_SHARED_LIBS=OFF
set BUILD_TEST=OFF
set BUILD_PYTHON=OFF
set CAFFE2_USE_MSVC_STATIC_RUNTIME=ON
set MSVC_Z7_OVERRIDE=OFF
set USE_CUDA=OFF

python ../tools/build_libtorch.py
</code></pre>
<p>but in the generated build directory, in the include folder, I don’t have any of the desired header files. Can I just copy over the header files from the downloaded shared library? Or is it more indicative of a building/compiling issue?</p>

username: jadenyouib
cooked: <p>Also, when I run cmake while setting the cmake prefix to be build inside build_libtorch, the variable<br>
<code>TORCH_INCLUDE_DIRS</code> is wrong. It is a made up include folder in the folder which I installed pytorch in.</p>

username: Hamza_Javaid
cooked: <aside class="quote no-group" data-username="jadenyouib" data-post="1" data-topic="221451" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/da6949/48.png" class="avatar"> jadenyouib:</div>
<blockquote>
<p>I am currently trying to build Torch as a static library with the following commands in Windows in the Visual Studio Native Tools Command Prompt.</p>
<pre><code class="lang-auto">mkdir build_libtorch
cd build_libtorch
set BUILD_SHARED_LIBS=OFF
set BUILD_TEST=OFF
set BUILD_PYTHON=OFF
set CAFFE2_USE_MSVC_STATIC_RUNTIME=ON
set MSVC_Z7_OVERRIDE=OFF
set USE_CUDA=OFF

python ../tools/build_libtorch.py
</code></pre>
<p>but in the generated build directory, in the include folder, I don’t have any of the desired header files. Can I just copy over the header files from the downloaded shared library? Or is it more indicative of a building/compiling issue?</p>
</blockquote>
</aside>
<p>Instead of <code>build_libtorch.py</code>, use CMake directly for better control:</p>
<pre><code class="lang-auto">mkdir build_static
cd build_static

cmake .. ^
  -DCMAKE_BUILD_TYPE=Release ^
  -DBUILD_SHARED_LIBS=OFF ^
  -DCMAKE_PREFIX_PATH="%CD%/../torch" ^
  -DPYTHON_EXECUTABLE="%CONDA_PREFIX%\python.exe" ^
  -DBUILD_TEST=OFF ^
  -DBUILD_PYTHON=OFF ^
  -DUSE_CUDA=OFF ^
  -DCAFFE2_USE_MSVC_STATIC_RUNTIME=ON ^
  -DCMAKE_INSTALL_PREFIX="%CD%/install"

cmake --build . --config Release --target install -- /m:%NUMBER_OF_PROCESSORS%
</code></pre>
Why is strict hardcoded to true in module._load_from_state_dict?

username: ekf
cooked: <p>Hello!</p>
<p>I’ve noticed that in module.py, the load_state_dict function fixes strict to True in the subsequent module._load_from_state_dict call as can be seen <a href="https://github.com/pytorch/pytorch/blob/51a708ffc679b13f99e4c7cf19bc00082a3266a6/torch/nn/modules/module.py#L2579" rel="noopener nofollow ugc">here</a>. This doesn’t seem to make sense to me, as this prevents everything downstream (e.g., hooks in _load_state_dict_pre_hooks) from being able to adapt to a non-strict load.</p>
<p>Could someone help me understand the motivation for this?</p>
Differentiating a method

username: Kaneda93100
cooked: <p>Hello, i’m new on pytorch. I would like to know if there is a way to differentiate the bisection<br>
method (for finding the roots of a certain function). Actually, it doesn’t seem differentiable for me, but is there a way to ensure that ? And in general, is there a way to know if a method is differentiable or not ?</p>
<p>Thanks for your attention.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Kaneda93100" data-post="1" data-topic="221586">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kaneda93100/48/77678_2.png" class="avatar"> Kaneda93100:</div>
<blockquote>
<p>And in general, is there a way to know if a method is differentiable or not ?</p>
</blockquote>
</aside>
<p>A simple way would be to check the <code>.grad_fn</code> of the output of an operation to see if a backward method is assigned to it:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(1, requires_grad=True)

y = x * 2
print(y.grad_fn)
# &lt;MulBackward0 object at 0x7f35a7f63b20&gt;

y = x ** 2
print(y.grad_fn)
# &lt;PowBackward0 object at 0x7f35686ebd60&gt;

y = x &gt; 0.1
print(y.grad_fn)
# None
</code></pre>
Autograd function returns tensor views

username: YangFei
cooked: <p>I have a very simple script</p>
<pre><code class="lang-auto">import torch

class PrintTensor(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input_tensor, tag):
        ctx.tag = tag  # Store tag as a class attribute instead
        print("input_tensor", input_tensor._base)
        return input_tensor

    @staticmethod
    def backward(ctx, grad_output):
        return grad_output, None

# Test code
model = torch.nn.Linear(4,4)
inp = torch.randn(4, 4)
print(inp._base)
output = model(inp)
print("output 1", output._base)
output = PrintTensor.apply(output, "test")
print("output 2", output._base)
loss = torch.sum(output)
loss.backward()
print(loss)
</code></pre>
<p>The output is</p>
<pre><code class="lang-auto">None
output 1 None
input_tensor None
output 2 tensor([[-0.7352,  0.6865,  0.5382, -0.0586],
        [ 0.8063, -0.1446,  1.1007, -0.0438],
        [ 0.1090, -0.5273,  0.3381, -0.9453],
        [-0.1048,  1.1854,  0.7933,  0.8607]], grad_fn=&lt;AddmmBackward0&gt;)
tensor(3.8586, grad_fn=&lt;SumBackward0&gt;)
</code></pre>
<p>After the autograd function of <code>PrintTensor</code> the output tensor becomes a tensor view of other tensor. Could someone help me understand this behavior? Thanks.</p>

username: soulitzer
cooked: <p>This is expected behavior. If a tensor is returned as-is, it is automatically turned into a view. This allows the custom autograd Function to attach a new grad_fn without overwriting the grad_fn of the input.</p>
Copying params between 2 identically sharded (FSDP) networks

username: Timofey_Abramski
cooked: <p>I’m fairly new to FSDP. My set up is that I have 2 networks, <code>model</code> and <code>avg_model</code>, where <code>avg_model</code> is updated to be an exponentially moving average (EMA) of <code>model</code>. Without FSDP, I can trivially duplicate the ema update on all gpus.</p>
<p>What I am trying to achieve, is to wrap both <code>model</code> and <code>avg_model</code> in FSDP (1) identically, and then in each training iteration only perform a local EMA update, where each gpu only updates its local portion of the <code>avg_model</code> using the local parameters of the <code>model</code> which should be possible due to both being sharded identically.</p>
<p>I have been able to shard the two networks identically by using the same fsdp config and wrapping policy. Within my training iteration, I need to perform an exponentially moving average (EMA) update of one network using the other, where my <code>ema_update</code> function looks roughly like this:</p>
<pre><code class="lang-auto">ema_update(model, avg_model, eta):
     avg_model_params = dict(get_unwrapped_model(avg_model).named_parameters())
     for param_name, model_param in model_params.items():
         avg_model_param.copy_(eta * avg_model_param + (1 - eta) * model_param)
</code></pre>
<p>Interestingly, when I run this <code>ema_update</code> function, along with a torch.distributed.barrier(), I don’t get the same results (within FSDP non-determinism), unless I run the following summon all params after my ema update:</p>
<pre><code class="lang-auto">ema_update(model, avg_model, eta)
torch.distributed.barrier()
with FSDP.summon_all_params(model, writeback=True, rank0_only=False):
    pass
</code></pre>
<p>It seems like the <code>summon_all_params</code> context manager should not be doing anything, apart from all gathering the parameters and then continuing, it does change the outcome of training and it ends up being necessary to achieve parity. Without it, it does appear as though the <code>avg_model</code> weights are being updated separately on each gpu, but the results/losses are slightly off from what I expect. How can this behaviour be explained?</p>
<p>It i worth bearing in mind that there are other parts to my training loop where the normal non-avged <code>model</code> is updated via a typical adam optimizer.</p>

username: bantmen
cooked: <p><a class="mention" href="/u/timofey_abramski">@Timofey_Abramski</a> did you figure out a solution?</p>

username: Timofey_Abramski
cooked: <p>No I haven’t, the best I have come up with is to do the apparently unnecessary</p>
<pre><code class="lang-auto">with FSDP.summon_all_params(model, writeback=True, rank0_only=False):
    pass
</code></pre>
<p>immediately after doing the local update, and that ensures I get the correct results. This feels kinda hacky, and seems unnecessary on top of the <code>torch.distributed.barrier()</code>, but without it I get incorrect results. It seems like there is some FSDP internal state that the summon forces to be correct, although not sure how this works internally.</p>
Sort YoloPose detection using Tracker ID

username: Miss_M
cooked: <p>Hi guys<br>
I am new to computer vision and tracking on PyTorch. I am using YOLO-Pose for animal pose estimation. YOLO estimates both bounding boxes and key points of the animal’s body.<br>
In each frame, there may be one or more animals, so I need to assign a unique ID to each using a tracker. Everything works fine when there is only one animal in the frame. However, when a second animal enters the frame, the model detects two animals, and I don’t know how to correctly arrange these detections and match them with the correct tracker IDs.<br>
Any advice would be really appreciated</p>

username: neonwatty
cooked: <p>Do you have a git repo you could share / code?  This would help others help you!</p>
Split backward into multiple gpus

username: mrk22
cooked: <p>Hi, I’m doing some experiments, and I got a huge computational graph, like 90GB. I’ve multiple GPUs and I would like to split the whole computational graph along them, how can I do that? Is there some framework that just changing my forward pass enables me to call the backward pass?</p>

username: valerian.rey
cooked: <p>I’m not sure it’s what you’re looking for, but maybe you can also reduce the batch size, so that the computational graph fits in the memory of a single GPU, and do distributed data parallel (DDP) so that your training is distributed across all of your GPUs. Here is a link to the <a href="https://docs.pytorch.org/tutorials/intermediate/ddp_tutorial.html#getting-started-with-distributed-data-parallel" rel="noopener nofollow ugc">documentation of distributed data parallel</a>. Note that it’s possible to combine this with what the other answers suggest (model parallelism): see <a href="https://docs.pytorch.org/tutorials/intermediate/ddp_tutorial.html#combining-ddp-with-model-parallelism" rel="noopener nofollow ugc">this</a>.</p>

username: mrk22
cooked: <aside class="quote no-group" data-username="james698henry" data-post="5" data-topic="221440">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/bcef8e/48.png" class="avatar"> james698henry:</div>
<blockquote>
<p>FSDP</p>
</blockquote>
</aside>
<p>Hi, thanks! It seems that zero/fsdp are what I’m looking for. Essentially, in my case, I need the gradients w.r.t. the input and I’ve the whole preprocessing + the forward pass of the model (I’m using the Hugging Face pretrained model, so I’m not modifying it) for which I’m keeping track of the computational graph (so it’s huge due to both the image size and the preprocessing). Is it possible to split the computation also in this case? Is there a way to launch the experiments without touching the model internals (since my model is downloaded from HF)?</p>
The shapes of output in .engine is 0, but the onnx works fine

username: Mahesh_Reddy_Tapeti
cooked: <p>I was trying to convert superpoint to a tensorrt engine, the onnx was working fine the input image returns the expected output. When trying to allocate buffers for the inputs, and outputs all the outpus shapes are 0 and that causes an error<br>
<code>Input 'image': shape = 1 \ Output 'keypoints': shape = 0 \ Output 'scores': shape = 0 \ Output 'descriptors': shape = 0 \</code></p>
<p>The error:<br>
LogicError: cuMemAlloc failed: invalid argument</p>
<p>This is how I converted the pytorch model to onnx<br>
torch.onnx.export(<br>
superpoint,<br>
image0,<br>
“superpoint.onnx”,<br>
input_names=[“image”],<br>
output_names=[“keypoints”, “scores”, “descriptors”],<br>
dynamic_axes={<br>
“image”: {0: “batch”, 2: “height”, 3: “width”},<br>
“keypoints”: {0: “batch”, 1: “num_points”},<br>
“scores”: {0: “batch”, 1: “num_points”},<br>
“descriptors”: {0: “batch”, 2: “num_points”}<br>
}<br>
)</p>
<p>conversion of onnx to tensorrt<br>
/usr/src/tensorrt/bin/trtexec --onnx=model.onnx --saveEngine=model.engine</p>
<p>How to fix it?</p>
Torchzero - modular optimization library

username: qq-me
cooked: <p>I made a library about optimization <a href="https://github.com/inikishev/torchzero" class="inline-onebox" rel="noopener nofollow ugc">GitHub - inikishev/torchzero: Modular optimization library for PyTorch.</a></p>
<p>The basic idea is that everything is a module and different modules can be combined, a basic example of Cautious Adam with gradient clipping</p>
<pre data-code-wrap="py"><code class="lang-py">opt = tz.Modular(
    model.parameters(),
    tz.m.ClipValue(4),
    tz.m.Adam(),
    tz.m.Cautious(),
    tz.m.LR(1e-3),
    tz.m.WarmupNormClip(steps=100),
)
</code></pre>
<p>but the modules can do a lot of things, like gradient approximation, line searches, for example here is  BFGS with approximated gradient and backtracking line search</p>
<pre data-code-wrap="py"><code class="lang-py">opt = tz.Modular(
    bench.parameters(),
    tz.m.FDM(),
    tz.m.BFGS(),
    tz.m.Backtracking(),
)
</code></pre>
<p>There is a module for sharpness-aware minimization which now can be used with any other optimizer (SAM, ESAM and MSAM), various preconditioning modules (Shampoo, SophiaG, AdaHessian, Full-matrix adagrad, a huge number of quasi-newton methods, newton, etc) that can also be applied to any other algorithm. Another thing that is possible is LoRA-like methods for low-memory training with any other optimizer, this is work in progress.</p>
<p>Algorithms can be combined in other ways too, like a sum <code>tz.m.Sum(tz.m.Adam(), tz.m.SOAP())</code> , grafting <code>tz.m.GraftModules(tz.m.Shampoo(), tz.m.RMSprop())</code> , or just put output of one optimizer into another one.</p>
<p>Besides the modularity a lot of the modules are algorithms that didn’t seem to have pytorch implementations available, especially a lot of quasi-newton methods.</p>
<p>A list of all modules is available at <a href="https://torchzero.readthedocs.io/en/latest/autoapi/torchzero/modules/index.html" rel="noopener nofollow ugc">https://torchzero.readthedocs.io/en/latest/autoapi/torchzero/modules/index.html</a>.</p>
<p>I will also add nonlinear least squares methods like gauss-newton soon, they can be integrated very easily, and will automatically support my trust region modules. Constrained optimization can also be added easily but I’m not motivated to work on it for now because I don’t have any uses for it. Let me know if there is anything else that is missing</p>
Create a torch tensor with a device pointer

username: hariIVI
cooked: <p>Is it possible to create a torch cuda float tensor from gpu memory?<br>
The following  code creates a numpy array and copy it to gpu memory. Now whether it is possible to create a torch tensor from gpu memory itself without copying it back to host as a numpy array. I know the example code doesn’t convey the actual purpose, in my application I do some computations on the inputs on gpu and then I would like to use one of the torch functionals on the output of the computations. Currently I copy initial results back to host and then create a torch tensor and move it back to the gpu. If I can directly create a cuda tensor directly, then I can avoid two copy operations.</p>
<pre><code class="lang-python">
from cuda import cudart, cuda
import numpy as np
import torch

# create a numpy array of shape (6, 3, 640, 640) with random values between 0 and 1 (float32) with C order
x = np.random.rand(6, 3, 640, 640).astype(np.float32, order="C")

# allocate memory on the GPU
x_gpu = cudart.cudaMalloc(x.nbytes)

# copy the numpy array to the GPU
cudart.cudaMemcpy(x_gpu, x, x.nbytes, cudart.cudaMemcpyKind.cudaMemcpyHostToDevice)

# create a cuda float tensor with same size as x
x_torch = torch.cuda.FloatTensor(x.size, device=torch.device('cuda'))

</code></pre>
<p>Will it be possible to copy the data from x_gpu to x_torch?</p>

username: ptrblck
cooked: <p>You could try to use <code>torch.utils.dlpack.from_dlpack(ext_tensor)</code> or <code>torch.frombuffer</code> but note that the object needs to implement the buffer protocol.</p>

username: youkaichao1
cooked: <p>I think <code>torch.frombuffer</code> does not work here. There’s an explicit warning:</p>
<blockquote>
<p>This function’s behavior is undefined when passed an object implementing the buffer protocol whose data is not on the CPU. Doing so is likely to cause a segmentation fault.</p>
</blockquote>
Using CUDA Graphs output during warmup/capture

username: kevmo314
cooked: <p>I’m looking at the <a href="https://docs.pytorch.org/docs/stable/notes/cuda.html#pytorch-api" rel="noopener nofollow ugc">API for CUDA graphs</a>, is it possible to use <code>static_output</code> from the warmup and capture phases? I tried this myself and am able to use the result out of the warmup call (unsurprisingly) but the capture phase seems to not produce the correct return value. Curiously, it seems to spit out a zero tensor but this zero tensor isn’t copied correctly to the static output?</p>
<p>My use case is I have a stateful function that I’d like to turn into a graph. Ideally I’d like to run the warmup and capture phase and be able to utilize the results before replaying the graph, as resetting the state is not ideal, but I haven’t been able to figure out why the call in the capture phase does not produce a valid output.</p>
<p>Or perhaps an alternate question: what is the return value in this line?</p>
<pre><code class="lang-auto">with torch.cuda.graph(g):
    static_output = static_input * 2
</code></pre>
<p>It differs from the output returned in the warmup call but it has no actual content…</p>

username: kevmo314
cooked: <p>Ah, classic. I figured this out right after posting despite spending a few hours on it before. The answer is I need to call <code>graph.replay()</code> once after doing the capture. This is <a href="https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#creating-a-graph-using-stream-capture" rel="noopener nofollow ugc">documented</a> as the work not actually being performed, and a related question for anyone else in the future: <a href="https://stackoverflow.com/questions/71618350/cuda-graph-problem-results-not-computed-for-the-first-iteration" rel="noopener nofollow ugc">https://stackoverflow.com/questions/71618350/cuda-graph-problem-results-not-computed-for-the-first-iteration</a></p>
Copy construct user warning despite using .clone().detach()

username: MSurtel
cooked: <p>Hello all,</p>
<p>I’ve provided a dummy version of some code I’ve been using:</p>
<pre><code class="lang-auto">def example_func(letters: Tuple[str, ...], length: int) -&gt; Tensor:
    letter_to_tensor: Dict[str, Tensor] = {
        'a': torch.ones(length),
        'b': torch.zeros(length),
        'c': torch.arange(length),
    }

    return torch.stack(
        tuple(torch.tensor(letter_to_tensor[letter].clone().detach()) for letter in letters),
        dim=1)

res = example_func(('c', 'b', 'a'), 50)

</code></pre>
<p>Upon running this code, I am met with the following user warning:<br>
"<br>
UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).<br>
tuple(torch.tensor(letter_to_tensor[letter].clone().detach()) for letter in letters),<br>
"</p>
<p>I was under the impression that using <code>letter_to_tensor[letter].clone().detach()</code> would prevent me from running into such a warning. How do I prevent this warning from cropping up?</p>
<p>Thanks in advance.</p>

username: MSurtel
cooked: <p>Bit of a silly oversight on my part, thank you for pointing it out.</p>
Question about ModuleList

username: the-ml-bulldog
cooked: <p>If I do something like <code>self.layers = nn.ModuleList([nn.Linear(3,3)] * 4)</code>, will each of those 4 linear layers have the same reference? Will gradient updates be the same for all 4 layers?</p>

username: paulk
cooked: <p>I think so. When you use <code>[nn.Linear(3,3)] * 4</code>, Python creates one <code>nn.Linear(3,3)</code> object, and then the list <code>[ ]</code> contains four references to that single object instead of creating four distinct <code>nn.Linear</code> instances. You can see this using <a href="https://www.w3schools.com/python/ref_func_id.asp" rel="noopener nofollow ugc">Pythons id() function</a> which returns a unique id for every object. Running</p>
<pre><code class="lang-auto">import torch.nn as nn

linear_layer = nn.Linear(3, 3)
layers_list = [linear_layer] * 4

print(id(layers_list[0]))
print(id(layers_list[1]))
print(id(layers_list[2]))
print(id(layers_list[3]))
</code></pre>
<p>returns the same id four times.<br>
Hope that helps.</p>
Question about utilizing vmap to calculate per-sample gradient

username: ymhwang
cooked: <p>Hi, there,<br>
I face a problem when I use vmap to calculate the batch of per-sample gradient of a RNN model. It always raise the dimension mismatch error and batching rule not implemented error. Is there any one can help me solve this?<br>
Here is my codes:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.func import functional_call, vmap, grad

class SimpleRNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super().__init__()
        self.hidden_size = hidden_size
        # batch_first=True means input/output is (batch, sequence, features)
        self.rnn = nn.RNN(input_size, hidden_size, batch_first=True)
        self.fc = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        # x: (sequence_length, input_size) for a single sequence
        # x: (batch_size, sequence_length, input_size) for a batch
        h0 = torch.zeros(1, x.size(0), self.hidden_size).to(x.device)
        out, _ = self.rnn(x, h0)
        output = self.fc(out[:, -1, :])
        return output

def compute_loss_for_single_sequence(model, single_input_sequence):
    input_with_batch = single_input_sequence.unsqueeze(0)
    loss = functional_call(model, dict(model.named_parameters()), input_with_batch)
    return loss

if __name__ == '__main__':


    # 1. Prepare batched data and model
    input_size = 10
    hidden_size = 20
    output_size = 1
    batch_size = 1000
    sequence_length = 5
    feature_dim = input_size
    model = SimpleRNN(input_size, hidden_size, output_size)
    params = {name: p for name, p in model.named_parameters()}
    # Input data: (batch_size, sequence_length, feature_dim)
    batch_input = torch.randn(batch_size, sequence_length, feature_dim, requires_grad=True)

    # 2. test batch evaluate the forward
    batch_output = vmap(compute_loss_for_single_sequence, in_dims=(None, 0))(model, batch_input)
    
    # 3. test batch per-sample gradients
    per_example_grad_fn = vmap(grad(compute_loss_for_single_sequence), in_dims=(None, 0))(model, batch_input)
</code></pre>
<ol>
<li>when test batch evaluate the forward, it raises the error:<br>
RuntimeError: Batching rule not implemented for aten::rnn_tanh.input. We could not generate a fallback.</li>
<li>when test batch per-sample gradients, it raises the error:<br>
I cannot correctly use vmap to evaluate function “compute_loss_for_single_sequence”, it obviously raises the error when test batch per-sample gradients.<br>
ValueError: Thing passed to transform API must be Tensor, got &lt;class ‘<strong>main</strong>.SimpleRNN’&gt;</li>
</ol>
Does INT8 Quantized Convolution Still Involve Floating-Point Operations？

username: witz
cooked: <p>Dear PyTorch community,</p>
<p>I have quantized a ResNet model to INT8 using ‟torch.quantization.convert()”. I noticed that a quantized convolution layer still keeps its scale parameter as a floating-point value. I think this scale is used to requantize the accumulated gemm output back to INT8. I would like to confirm:</p>
<p>Does the quantized convolution operator perform any floating-point computations internally, or is the entire operation carried out in pure INT8/INT32 arithmetic?</p>
<p>Thank you!</p>

username: William_Gram
cooked: <p>Yes, INT8 convolution may still use the floating point for a scaling and  the activation functions.</p>

username: witz
cooked: <p>Do you mean that during the requantization from int32 to int8 we multiply by an FP32 scaling factor rather than rely on bit-shifting?</p>
PyTorch compute compatibility with sm_120

username: Manushree
cooked: <p>I am using RTX5090 with sm_120, however the CUDA compute compatibility supports only upto sm_90, can I fallback to use sm_90 on sm_120 without having to wait for Pytorch to make updates on this. Am training my YOLO model with CPU despite having a high-end system. FYI I have even tried using the nightly version instead of the stable one. Any fixes ?</p>

username: ptrblck
cooked: <p>All of our stable and nightly binaries built with CUDA 12.8 already support Blackwell architectures including the RTX 5090.</p>
Create PackedSequence directly

username: alexj
cooked: <p>Hi, I have sequences that I need to GRU with big head and long tail: the maximum is high, but many sequences have  a couple of entries max. So while the sequ3ences in a single tensor are  manageable, If I create a padded tensor, it would become 10x bigger and break my puny GPU.<br>
I want to create rnn.PackedSequence without going through padded, walking straight into “Instances of this class should never be created manually”. That is exactly what I need to do.<br>
I created and example that works perfectly:</p>
<pre><code class="lang-auto">t = torch.tensor([1, 2, 3, 1, 2, 3, 1, 2]).float()
lengths = torch.tensor([3, 3, 2], dtype=torch.int64)
t = t.unsqueeze(1)
packed = torch.nn.utils.rnn.PackedSequence(t, batch_sizes=lengths)
packed
thegru = torch.nn.GRU(1, 1, batch_first=True)
res, final = thegru(packed)
</code></pre>
<p>It shows exactly what I wanted. However when I run it with real data (about a million entries with about 20 channes, representing about 100,000 batches), the kernel crashes<br>
HELP!</p>
Problems with RTX5090 in compatibility with PyTorch

username: Botd
cooked: <p>When training the neural network models based on PyTorch, the RTX5090(Stable 2.7.1 / Preview (Nightly) ) sometimes performs less efficiently than the RTX4090. The testing code is shown as follow. When testing the Big data, RTX5090 performs less efficiently than the RTX4090. But when testing the Small data, RTX5090 performs more efficiently than the RTX4090. I am confused by this phenomenon.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim
import time

#(1)Small data
# x = torch.rand((10000,3))
# y = torch.rand((10000,1))

#(2)Big data
x = torch.rand((100000,3))
y = torch.rand((100000,1))

class Net(nn.Module):
    def __init__(self,):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(3, 200)
        self.fc2 = nn.Linear(200, 200)
        self.fc3 = nn.Linear(200, 200)
        self.fc4 = nn.Linear(200, 200)
        self.fc5 = nn.Linear(200, 1)

    def forward(self, x):
        x = torch.sin(self.fc1(x))
        x = torch.sin(self.fc2(x))
        x = torch.sin(self.fc3(x))
        x = torch.sin(self.fc4(x))
        x = self.fc5(x)
        return x

T1=time.time()
model = Net()
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)

for epoch in range(300):
    optimizer.zero_grad()
    outputs = model(x)
    loss = criterion(outputs, y)
    loss.backward()
    optimizer.step()

    if epoch % 100 == 0:
        print(f'Epoch {epoch}, Loss: {loss.item():.4f}')

T2=time.time()
print(T2-T1)

</code></pre>

username: ptrblck
cooked: <p>You could profile the end2end use case with e.g. Nsight Systems to narrow down the bottlenecks in your workloads as they could come from e.g. the data loading and processing stage and not the GPU computation workloads (since small datasets perform well on the 5090 vs. 4090).</p>

username: Botd
cooked: <p>Thank you for your prompt reply. First, in Preview (Nightly), I conducted several tests and found that for convolutional neural networks, 5090 had a significant advantage over 4090. However, when training fully connected neural networks, especially for large datasets, 5090 seemed to perform less well than 4090. The data was not loaded or preprocessed, but was directly generated using torch.rand.</p>
Tensor chunking

username: Ravi_Ghadia
cooked: <p>Is there an easy way to perform tensor chunking similar to <code>torch.chunk</code> but make it return different tensors (instead of different views of the same tensor).</p>
<p>I want to have seperate memory references for individual chunks, since I wish to offload some specific chunks onto the CPU.</p>
<p>Is there an efficient way to do this without performing <code>clone</code> on every chunk?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Ravi_Ghadia" data-post="1" data-topic="221420">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ravi_ghadia/48/77642_2.png" class="avatar"> Ravi_Ghadia:</div>
<blockquote>
<p>I want to have seperate memory references for individual chunks, since I wish to offload some specific chunks onto the CPU.</p>
</blockquote>
</aside>
<p>Wouldn’t a <code>to("cpu")</code> on the corresponding chunk just work? This should copy the chunk to the CPU without cloning it on the GPU beforehand.</p>

username: Ravi_Ghadia
cooked: <p>Yes, but that won’t allow offloading it from GPU. I wish to clear up the corresponding memory from the GPU. So <code>chunks[0].to(cpu)</code> will essentially copy <code>chunks[0]</code> data to the CPU, but the original tensor as a whole (<code>chunks</code>) still remains on the GPU, right?</p>
How to Efficiently Gather Python Objects Across GPUs Without GPU-to-CPU-to-GPU-to-CPU Overhead in torch.distributed?

username: 5o1
cooked: <p>I am running a distributed inference task where each GPU predicts one frame of an image per sample. At the end of each epoch, I need to gather all the predicted frames from all GPU processes into a single tensor. Finally, I will save the gathered frames to a file.</p>
<p>I came across <code>torch.distributed.all_gather_object()</code> to achieve this, but the documentation states the following warnings:</p>
<ol>
<li><strong>Note</strong><br>
For NCCL-based processed groups, internal tensor representations of objects must be moved to the GPU device before communication takes place. In this case, the device used is given by <code>torch.cuda.current_device()</code> and it is the user’s responsiblity to ensure that this is set so that each rank has an individual GPU, via <code>torch.cuda.set_device()</code>.</li>
<li><strong>Warning</strong><br>
Calling <code>all_gather_object()</code> with GPU tensors is not well supported and inefficient as it incurs GPU → CPU transfer since tensors would be pickled. Please consider using <code>all_gather()</code> instead.</li>
</ol>
<p>If I use <code>all_gather_object()</code> as follows:</p>
<ul>
<li>GPU to CPU transfer occurs.</li>
<li>Pickling and gathering happens.</li>
<li>CPU to GPU transfer occurs after gathering.</li>
</ul>
<p>After this, to save the gathered results to a file, I will need another:</p>
<ul>
<li>GPU to CPU transfer.</li>
</ul>
<p>The final step introduces redundant operations, and transferring tensors back to the GPU after gathering increases the risk of out-of-memory (OOM) errors.</p>
<h3><a name="p-472537-my-goals-1" class="anchor" href="#p-472537-my-goals-1"></a>My Goals:</h3>
<ol>
<li>Since my prediction results are stored in a dictionary with a key-value structure like <code>{filename: (frame_idx, tensor)}</code>, I need to gather arbitrary Python objects (at least dictionaries).</li>
<li>I cannot ensure that frames distributed to each GPU process are synchronized or evenly distributed across samples. Hence, using <code>all_gather()</code> with tensors directly would lead to synchronization issues.</li>
<li>I want to avoid transferring tensors back to the GPU after gathering, as this can cause additional memory overhead and potential OOM issues.</li>
</ol>
<h3><a name="p-472537-question-2" class="anchor" href="#p-472537-question-2"></a>Question:</h3>
<p>What is the best approach to gather arbitrary Python objects (e.g., dictionaries) across <strong>ALL</strong> GPU processes <strong>to ONLY ONE CPU process</strong>  efficiently without incurring unnecessary GPU-to-CPU-to-GPU-to-CPU transfers?</p>

username: 5o1
cooked: <p>pin:</p>
<p>Now I found the torch kit <code>TensorDict</code>. According to its documentation, it can be used to exchange data across processes in a distributed environment. I’m ready to see if it works.</p>
Does the optimizer have to be created after model is moved to the GPU?

username: ening
cooked: <h2><a name="p-472700-codes-1" class="anchor" href="#p-472700-codes-1"></a>codes</h2>
<p>model = MllamaForConditionalGeneration.from_pretrained(<br>
MODEL_NAME,<br>
attn_implementation=“sdpa”,<br>
device_map=“cpu”,<br>
torch_dtype=torch.float32,<br>
)<br>
optim = AdamW(model.parameters(), lr=1e-5)<br>
model.to(“cuda:0”)</p>
<h2><a name="p-472700-questions-2" class="anchor" href="#p-472700-questions-2"></a>Questions</h2>
<p>In the codes above, I create <strong>mllama</strong> model on cpu and use its parameters to create optimizer, then move model to cuda.<br>
Is this approach feasible?<br>
Will there be any problems with updating the model parameters?</p>

username: ptrblck
cooked: <p>Parameters are moved inplace so creating the optimizer before moving the model should be fine.<br>
This simple code snippet also shows it:</p>
<pre data-code-wrap="python"><code class="lang-python">model = torch.nn.Linear(10, 10)
optimizer = torch.optim.Adam(model.parameters())
print([p.device for p in optimizer.param_groups[0]['params']])
# [device(type='cpu'), device(type='cpu')]

model.cuda()
print([p.device for p in optimizer.param_groups[0]['params']])
# [device(type='cuda', index=0), device(type='cuda', index=0)]
</code></pre>

username: ening
cooked: <p>I understand! Thanks a lot!</p>

username: ptrblck
cooked: <p>Did you see any issues or is this a general question?<br>
Generally, I would still recommend finishing the actual model setup before creating the optimizer, especially if you are applying more advanced model sharding approaches.</p>

username: ening
cooked: <p>This problem typically occurs when using <strong>FSDP</strong> or <strong>DeepSpeed</strong> to accelerate large model training. When using <code>transformers</code> library, it is easy to set optimizer config to create <code>optimizer</code>. However, if using own custom library for training, we usually pass directly <code>model</code>, <code>optimizer</code> and <code>lr_scheduler</code> to <code>SFT Interface</code>. In this case, model will be processed, such as: <code>Move to Distrubuted GPUs and use FSDP or DeepSpeed Wrap</code>. So we hope that the model location is on the <code>CPU</code> and using the model parameters to create optimizer.</p>
Why is AdamW not the default weight decay implementation in pytorch?

username: sad_robot
cooked: <p>I noticed that the default torch.optim.Adam() optimizer has a weight_decay=0 hyper parameter, yet torch.optim.AdamW is a separate implementation (why not replace the original?). Additionally torch.optim.RAdam() optimizer has a weight_decay=0 and a decoupled_weight_decay=False hyper parameter.</p>
<p>Both of these design choices seem to indicate that AdamW’s decoupled weight decay implementation is only situationally useful and that the original “coupled weight decay” algorithm is a better default choice for most use cases. Which obviously directly contradicts the AdamW paper.</p>
<p>I’m wondering if there were later research results that gave the Pytorch designers this impression? Or if perhaps there have been issues reproducing AdamW results or something?<br>
Intuitively I don’t understand why you would couple momentum with a regularization term.</p>

username: ptrblck
cooked: <p>A lot of defaults are kept for backward compatibility reasons and you would sometimes see that higher-level API libs are changing these to the current SOTA usage.<br>
I haven’t checked the details here but this could also be the case for potentially sub-optimal arguments used in these optimizers.</p>
Using multiple gpus (ddp pytorch) and showing loss graphs by dis.reduce

username: SM2023
cooked: <p>Hi All, I am using ddp pytorch for fine tunning my model. I am wondering how I can save the  average of loss function from all gpus for showing the loss graph. when I printing the loss in the code, it shows me three losses from 3 gpus which make sense. but for graph I need to reduce the loss is the following code correct to apply? is the definition of  “avg_train_loss_reduced” correct to use as a final loss for graph? many thanks for your feedback.</p>
<pre><code class="lang-auto">    model = copy.deepcopy(model_or)

    model=model.to(gpu_id)
    model = DDP(model, device_ids=[gpu_id])
    print("gpu_id",gpu_id)
    # ========================================
    #               Training
    # ========================================
          
    for epoch_i in range(0, total_epochs):
        
        print("")
        print('======== Epoch {:} / {:} ========'.format(epoch_i + 1, total_epochs))
        print('Training...')

        ##########################################
        train_loader.sampler.set_epoch(epoch_i)
        b_sz = len(next(iter(train_loader))[0])
        print(f"[GPU{gpu_id}] Epoch {epoch_i} | Batchsize: {b_sz} | Steps: {len(train_loader)}")
        train_loader.sampler.set_epoch(epoch_i)
        ##########################################
        total_train_loss = 0

        model.train()

        for step, batch in enumerate(train_loader):
            print("len(train_loader)",len(train_loader))
            #################################
            b_input_ids = batch[0].to(gpu_id,non_blocking=True)
            b_labels = batch[0].to(gpu_id,non_blocking=True)
            b_masks = batch[1].to(gpu_id,non_blocking=True)
            #################################

            optimizer.zero_grad()        

            outputs = model(  b_input_ids,
                             labels=b_labels, 
                              attention_mask = b_masks,
                              token_type_ids=None
                            )

            loss = outputs[0]  
            batch_loss = loss.item()
            total_train_loss += batch_loss
            loss.backward()
            optimizer.step()
            scheduler.step()
        
         # Calculate the average loss over all of the batches.

        avg_train_loss = total_train_loss / len(train_loader)  

        ## reduce the average loss from 3 gpus  and get sum
        dist.reduce(avg_train_loss, 0, op=dist.ReduceOp.SUM)
        
        ### devide by 3 because I used 3 gpus
        avg_train_loss_reduced=avg_train_loss/3

        Path_3=pt_save_directory+'/'+'avg_train_loss_reduced='+str(gpu_id)+".csv"
        torch.save(avg_train_loss_reduced,Path_3)
</code></pre>

username: KL1236
cooked: <p>Hi! I was wondering if you have any update about this issue?</p>
Very slow training with nn.Embedding

username: ptrch_c_m
cooked: <p>I use <code>nn.Embedding</code> initialized with GloVe embeddings and I want to further train the embeddings when I fine tune on my task. The GloVe embeddings are a bit more 2.1M, and their dimension is 300. The training becomes extremely slow. I think that this might be due to the size of the <code>Embedding</code> table. I don’t know if this is relatable, but I have small relatively small sentences (many of them might be less than 15 words). The weights of <code>Embedding</code> change only for the words fount in the batch right? Is this very slow training normal?</p>

username: vdw
cooked: <p>Do you really have a vocabulary size of 2.1 million? Or do you load all pretrained vectors into your embedding layer?</p>
<p>You only need the vectors for the words in your vocabulary? So if you have, say, 100k words/tokens, the input size of the embedding layer only needs to be 100k.</p>
<p>Even if your whole vocabulary is 2.1 million tokens, try considering only the, say, 100k most frequent ones. Due to Zipf’s Law, the 100k most frequent tokens probably cover 98% of your dataset.</p>

username: ptrch_c_m
cooked: <p>Thank you, you are right, that was it! Now the training is very fast.</p>