akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Error generating Grad-CAM: 'tuple' object has no attribute 'cpu'

NITISH_KUMAR2: <p>help to sovle this error: Error generating Grad-CAM: ‘tuple’ object has no attribute ‘cpu’</p>
<h1><a name="p-471932-add-this-class-somewhere-above-your-grad-cam-code-1" class="anchor" href="#p-471932-add-this-class-somewhere-above-your-grad-cam-code-1"></a>Add this class somewhere above your Grad-CAM code</h1>
<p>class GradCAMWrapper(nn.Module):<br>
def <strong>init</strong>(self, model, model_name):<br>
super().<strong>init</strong>()<br>
self.model = model<br>
self.model_name = model_name</p>
<pre><code>def forward(self, x):
    # For ViT, output is a tuple; for hybrid, output is tensor
    out = self.model(x)
    if isinstance(out, tuple):
        return out[0]  # or out.logits if available
    if hasattr(out, "logits"):
        return out.logits
    return out
</code></pre>
<p><span class="hashtag-raw">#-----------------------------</span></p>
<p>def find_first_tensor(x):<br>
“”“Recursively find the first torch.Tensor in a nested tuple/list structure.”“”<br>
if isinstance(x, torch.Tensor):<br>
return x<br>
elif isinstance(x, (tuple, list)) and len(x) &gt; 0:<br>
return find_first_tensor(x[0])<br>
else:<br>
raise ValueError(“No tensor found in Grad-CAM output”)</p>
<p>def visualize_gradcam(model, image, device, model_name=‘vit’):<br>
model.eval()<br>
wrapped_model = GradCAMWrapper(model, model_name)</p>
<pre><code>if model_name == 'vit':
    target_layers = [model.vit.encoder.layer[-1]]
elif model_name == 'hybrid':
    target_layers = [model.vit.encoder.layer[-1]]
else:
    logging.error(f"Unsupported model_name: {model_name}")
    return None

try:
    cam = GradCAM(model=wrapped_model, target_layers=target_layers)
except Exception as e:
    logging.error(f"Error initializing GradCAM: {e}")
    return None

input_tensor = image.unsqueeze(0).to(device)

try:
    grayscale_cam = cam(input_tensor=input_tensor, targets=None)
    logging.info(f"Type of grayscale_cam after cam(): {type(grayscale_cam)}")
    logging.info(f"Value of grayscale_cam after cam(): {grayscale_cam}")
    grayscale_cam = find_first_tensor(grayscale_cam)  # &lt;-- This line is key!
    grayscale_cam = grayscale_cam.detach().cpu().numpy()
    if not isinstance(grayscale_cam, np.ndarray):
        grayscale_cam = np.array(grayscale_cam)
except Exception as e:
    logging.error(f"Error generating Grad-CAM: {e}")
    return None

img = image.permute(1, 2, 0).cpu().numpy()
img = img * np.array([0.229, 0.224, 0.225]) + np.array([0.485, 0.456, 0.406])
img = np.clip(img, 0, 1)

try:
    visualization = show_cam_on_image(img, grayscale_cam, use_rgb=True)
    return visualization
except Exception as e:
    logging.error(f"Error creating Grad-CAM visualization: {e}")
    return None
</code></pre>
Distributed learning in windows

reinforced: <p>Is it unfeasible to do distributed learning in windows</p>
<p>I want to do distributed learning by connecting two windows computers. Each computer has one gpu. But seems it does not work. Looks like process does not work in set up part which means computer does not handle</p>
<pre><code class="lang-auto">dist.init_process_group(
 backend="gloo",
...
) 
</code></pre>
<p>this part.</p>
Resetting internal state of optimizer

Matthias_Moller: <p>I would like to reset the internal state of the optimizer after a couple of epochs. I figured out that I can do</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">opt.state().clear();
</code></pre>
<p>but afterwards the call</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">opt.step(closure);
</code></pre>
<p>with <code>closure</code> being defined as</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">auto closure = [&amp;]() {
        // Reset gradients
        net_-&gt;zero_grad();

        // Execute the model on the inputs
        outputs = net_-&gt;forward(inputs);

        // Compute the loss value
        loss = this-&gt;loss(outputs, epoch);

        // Compute gradients of the loss w.r.t. the model parameters
        loss.backward({}, true, false);

        return loss;
      };
</code></pre>
<p>gives an immediate segmentation fault.</p>
<p>Any help is appreciated.</p>
Tensor parallel numeric mismatch

oahz48: <p>Hi, I am applying the tensor parallel to a submodule</p>
<pre><code class="lang-auto">self.ffn = nn.Sequential(
    nn.Linear(dim, ffn_dim),
    nn.GELU(approximate="tanh"),
    nn.Linear(ffn_dim, dim),
)
</code></pre>
<p>the parallel plan is</p>
<pre><code class="lang-auto">"ffn": PrepareModuleInput(
    input_layouts=(Replicate(),),
    desired_input_layouts=(Replicate(),),
),
"ffn.0": ColwiseParallel(),
"ffn.2": RowwiseParallel(
    output_layouts=Replicate(),
    use_local_output=True,
),
</code></pre>
<p>But after the parallel, the results are not numeric matched. e.g., compute the output norm is 73.18 vs. 73.15. Is this expected or is there something wrong? Thanks.</p>

tianyu: <p>what are the precisions you use with / without TP? Different dtypes in communication / computation could cause different numerics.</p>
Can GradScaler and autocast be safely unified with enabled flag for both AMP and non-AMP training

mw-v: <p>Hi everyone,</p>
<p>I’m currently maintaining two separate training loops for my PyTorch model: one standard (float32) and one using automatic mixed precision (AMP) with <code>torch.amp</code>.</p>
<p>To simplify my codebase, I would like to merge them into a single training loop using the <code>enabled</code> flag for both <code>torch.amp.GradScaler</code> and <code>torch.autocast</code>.</p>
<p>Here’s my question:</p>
<p>If I use this pattern:</p>
<pre data-code-wrap="python"><code class="lang-python">scaler = torch.amp.GradScaler(enabled=config.enable_amp)

for x, y in train_loader:
    with torch.autocast(device_type=device.type, dtype=torch.float16, enabled=config.enable_amp):
        logits = model(x)
        loss = criterion(logits, y)

    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()
    optimizer.zero_grad()
</code></pre>
<p>When <code>config.enable_amp</code> is <code>False</code>, will the behavior be exactly equivalent to standard full-precision training, i.e.:</p>
<ul>
<li><code>scaler.scale(loss).backward()</code> → behaves like <code>loss.backward()</code></li>
<li><code>scaler.step(optimizer)</code> → behaves like <code>optimizer.step()</code></li>
<li><code>scaler.update()</code> → becomes a no-op</li>
</ul>
<p>Is this the correct and safe way to generalize the training loop for both AMP and non-AMP modes without duplicating logic?</p>
<p>Thanks in advance!</p>

ptrblck: <p>Yes, this should be the case if you properly pass the <code>enabled</code> argument to <code>GradScaler</code> and <code>autocast</code>.</p>
<p>E.g. <a href="https://github.com/pytorch/pytorch/blob/b020971e7806bba39aecf636e59e743911831ad8/torch/amp/grad_scaler.py#L206-L207"><code>scale()</code></a> will just return the passed <code>outputs</code> and will be a no-op, <a href="https://github.com/pytorch/pytorch/blob/b020971e7806bba39aecf636e59e743911831ad8/torch/amp/grad_scaler.py#L387-L388"><code>step()</code></a> will just call <code>optimizer.step()</code> is it’s not enabled, etc.</p>

mw-v: <p>Also, a quick follow-up question:</p>
<p>Is it OK if my validation loop <strong>does not</strong> use mixed precision (i.e., runs in full float32), even when training uses AMP? Or should I also wrap the validation step in:</p>
<pre data-code-wrap="python"><code class="lang-python">with torch.autocast(device_type=device.type, dtype=torch.float16, enabled=config.enable_amp):
    logits = model(x)
    loss = criterion(logits, y)
</code></pre>
<p>The reason I ask is because I use the validation loss for early stopping, so I want to make sure that not using AMP in validation won’t invalidate or misalign the early stopping behavior.</p>
<p>Thanks again!</p>

ptrblck: <p>Yes, you can use FP32 in the validation loop if needed.</p>
Static build that supports torchscript

ionilith: <p>I made a custom build of Torch to link into an existing c++ application.  This is the simplest build of torch that I could manage.  To give you background, this is for a POS model that has already been trained but only needs to be implemented in the c++ program.  When loading the model into the program I get the following error …<br>
Unknown builtin op: aten::mul.<br>
Could not find any similar ops to aten::mul. This op may not exist or may not be currently supported in TorchScript.<br>
:<br>
File “”, line 3</p>
<p>def mul(a : float, b : Tensor) → Tensor:<br>
return b * a<br>
~~~~~ &lt;— HERE<br>
def add(a : float, b : Tensor) → Tensor:<br>
return b + a<br>
‘mul’ is being compiled since it was called from ‘gelu_0_9’<br>
File “”, line 3</p>
<p>def gelu_0_9(self: Tensor) → Tensor:<br>
return torch.gelu(self, approximate=‘none’)<br>
~~~~~~ &lt;— HERE</p>
<p>As far as I can tell what this means I need to force the entire object file to be linked into my binary.  That isn’t a problem. The problem is I don’t know which library has the aten::mul (and other)_operators in it.  The compile I options I am using are as follows…</p>
<p>cmake ../ -DUSE_CUDA=OFF -DUSE_FBGEMM=OFF -DUSE_KINETO=OFF -DUSE_CUPTI_SO=OFF -DUSE_OPENMP=OFF -DUSE_NUMPY=OFF -DB    UILD_SHARED_LIBS=OFF -DBUILD_CUSTOM_PROTOBUF=ON -DUSE_NNPACK=ON -DUSE_QNNPACK=OFF -DUSE_PYTORCH_QNNPACK=OFF -DUSE_    XNNPACK=OFF -DUSE_DISTRIBUTED=OFF -DUSE_MKLDNN=OFF -DBUILD_PYTHON=OFF -DBUILD_CAFFE2=OFF -DUSE_NCCL=OFF -DUSE_VALG    RIND=OFF -DONNX_ML=OFF -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=../../install</p>
<p>So everything is off.  There maybe something I have turned off I need to support torchscript but for the life of me I can’t find figure out what it is.  Any help with either the compile or which libraries are necessary to make this work would be very helpful. Thank you.</p>

ionilith: <p>I figured it out.  So for anyone who runs into this issue the first thing you are going to need to do is for the entire libtorch_cpu(or whatever gpu you are using) static library to be loaded into your binary.  You can do this with  ld using -Wl --whole-archive,.  After you do that you will get a ton of linking errors.</p>
<p>Now when you use cmake --install .  it quite sadly only puts some of the libraries that you need in the install path.  In truth you are going to need nearly all of them.  They will be in your cmake build directory/lib/  Here is the order I am linking them (order matters).<br>
‘torch_cpu’,<br>
‘torch’,<br>
‘c10’,<br>
‘protobuf’,<br>
‘onnx’,<br>
‘onnx_proto’,<br>
‘cpuinfo’,<br>
‘nnpack’,<br>
‘pthreadpool’,<br>
‘Caffe2_perfkernels_avx2’,<br>
‘Caffe2_perfkernels_avx’,<br>
‘sleef’,<br>
‘ittnotify’</p>
<p>Important note on sleef.  It is not in the /lib directory.  It is however in /sleef/lib so be sure to grab it out of there.  With that I have a static cpu build with TorchScript running in a c++ application.  Enjoy.</p>
How pytorch treat with inplace operation in backward

skyevilman: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c5bcf074038a12b72bccfdc4896e8573245d26f.png" data-download-href="https://discuss.pytorch.org/uploads/default/5c5bcf074038a12b72bccfdc4896e8573245d26f" title="Snipaste_2025-06-16_14-31-17"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5c5bcf074038a12b72bccfdc4896e8573245d26f.png" alt="Snipaste_2025-06-16_14-31-17" data-base62-sha1="db2CQm2N8Fp8hmUhdAe3cF48ROv" width="513" height="402"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Snipaste_2025-06-16_14-31-17</span><span class="informations">513×402 3.77 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The code segment above shows a forward and a backward with a mul_ inplace operation.We can see before inplace op, c1 is [3,6,9], and after inplace op, c2 is [6, 24, 54].when backward is triggered,<br>
how the MulBackward0 func to compute grad for  c1, however now c1 has been changed to c2. If MulBackward0 func use the c2 value to compute grad for b, then the result will not be [5, 10, 15] shown above,otherwise there is some reserved mechanism for original c1? please give some ideas.</p>

soulitzer: <p>If the original input of the in-place is needed for backward, autograd will clone that input.</p>
Torch.cuda.is_available() returns Fasle in CUDA12.6

Joey2: <p>Could you troubleshoot this issue please? I checked GPU and CUDA driver, Pytorch version are installed properly for CUDA 12.6</p>
<p>nvcc --version</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Sep_12_02:18:05_PDT_2024
Cuda compilation tools, release 12.6, V12.6.77
Build cuda_12.6.r12.6/compiler.34841621_0
</code></pre>
<p>sudo nvidia-smi</p>
<pre><code class="lang-auto">Tue Jun 17 13:37:39 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03              Driver Version: 560.35.03      CUDA Version: 12.6     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla V100-PCIE-16GB           Off |   00000000:3B:00.0 Off |                    0 |
| N/A   30C    P0             23W /  250W |       4MiB /  16384MiB |      0%      Default |
|                                         |                        |                  N/A |
</code></pre>
<p>python</p>
<pre><code class="lang-auto">Python 3.11.9 (main, Jun 17 2025, 11:22:18) [GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
torch.cuda.is&gt;&gt;&gt; torch.cuda.is_available()
False
&gt;&gt;&gt; torch.cuda.device_count()
/data/i571991/.pyenv/versions/3.11.9/lib/python3.11/site-packages/torch/cuda/__init__.py:789: UserWarning: Can't initialize NVML
  warnings.warn("Can't initialize NVML")
0
&gt;&gt;&gt; torch.__version__
'2.7.1+cu126'
&gt;&gt;&gt; torch.version.cuda
'12.6'
</code></pre>
<p>uname -a<br>
<code>Linux ccde2gl1795 5.14.21-150500.55.100-default #1 SMP PREEMPT_DYNAMIC Thu Apr 3 17:03:15 UTC 2025 (a9fe8f3) x86_64 x86_64 x86_64 GNU/Linux</code></p>

ptrblck: <aside class="quote no-group" data-username="Joey2" data-post="1" data-topic="220864">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/joey2/48/77391_2.png" class="avatar"> Joey2:</div>
<blockquote>
<p><code>Can't initialize NVML</code></p>
</blockquote>
</aside>
<p>Are you able to initialize any CUDA sample as it seems your driver wasn’t properly installed (or you did not reboot after updating it)?</p>
Function type in TorchScript

vadimkantorov: <p>I’d like to parametrize my <code>torch.jit.script</code>'ed function with a function argument, i.e. whether a Forward-Backward algorithm should use <code>lambda x: torch.max(x, dim = dim).values</code> or <code>torch.logsumexp(x, dim = dim)</code>. When I pass it as lambda, TorchScript complains that I’m calling a tensor-typed value which happens because it types the argument as Tensor (despite the fact that it’s initialized to a lambda by default).</p>
<p>Is there any way to tell TorchScript to type a value as a Callable?</p>
<p>Is it possible to template a TorchScript function by an external callable values via some other mechanism?</p>
<p>Should I create an issue about this on GitHub?</p>

driazati: <p>TorchScript currently doesn’t support callables as values (we’re working on supporting it in the coming months). The compiler doesn’t look at the default values at all since it’s not always possible to recover a full type from the default, so if types aren’t specified it just assumes everything is a Tensor. To specify otherwise, you can use Python 3 type hints or mypy style type comments (<a href="https://pytorch.org/docs/master/jit_language_reference.html#supported-type">details</a>).</p>
<p>Function attributes on modules is the closest you can get at the moment, so something like</p>
<pre><code class="lang-python">def my_fn(x, some_callable):
    return some_callable(x + 10)
</code></pre>
<p>would have to be changed something like to</p>
<pre><code class="lang-python">class M(nn.Module):
    def __init__(self, some_callable):
        self.some_callable = some_callable
    
    def forward(self, x):
        return self.some_callable(x + 10)

torch.jit.script(M(the_function))
torch.jit.script(M(some_other_function))
</code></pre>

Enamex: <p>This still has some big unfortunate limitations.</p>
<p>You can’t store some callables in a <code>ModuleList</code> or a <code>ModuleDict</code> and index that, because TorchScript sees them as modules and cannot subscript them.</p>

janekl: <p>In my use case a function complied by torch.jit.scripts decorator is about 30% faster which is very nice performance boost! But I need to give up on its flexibility and freeze the function used as an argument to the decorated function (which is OK for when it comes to deployment where such flexibility is not needed as opposed to running many different experiments in development phase).</p>
<p>So +1 from me to enable <code>Callable</code> as an argument when using torch.jit.scripts so that it can be directly used without any extra effort like in the solution proposed.</p>

jackshi: <p>This is also very important because torch flex attention utilizes functions for <code>score_mod</code> and <code>mask_mod</code>. At the moment flex attention is basically not possible for torchscript tracing.</p>
TorchDispatchMode generates aten::detach for activations like ReLU

99991: <p>When inspecting the called PyTorch functions with TorchDispatchMode, I sometimes get <code>aten::detach</code> (depending on the model), which is not documented here: <a href="https://docs.pytorch.org/docs/stable/torch.compiler_ir.html" class="inline-onebox" rel="noopener nofollow ugc">IRs — PyTorch 2.7 documentation</a></p>
<p>Reproducer:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
from torch.utils._python_dispatch import TorchDispatchMode

class MyDispatch(TorchDispatchMode, list):
    def __torch_dispatch__(self, func, types, args, kwargs=None):
        print(func.name())
        return func(*args, **kwargs)

model = nn.Sequential(nn.Conv2d(1, 1, 1), nn.ReLU())

x = torch.rand(1, 1, 32, 32)

with MyDispatch():
    model(x)
</code></pre>
<p>My questions:</p>
<ol>
<li>What is <code>aten::detach</code>?</li>
<li>Why does this code call <code>aten::detach</code>?</li>
<li>Why does the code not call <code>aten::detach</code> for other models, for example <code>model = nn.Sequential(nn.ReLU(), nn.Conv2d(1, 1, 1))</code> where the order of <code>ReLU</code> and <code>Conv2d</code> is reversed compared to the reproducer model or when building my own <code>ReLU</code> with <code>torch.maximum</code>?</li>
</ol>
Building from GIT error: PyTorch needs CUDNN-8.5 or above but found 8.2.4

vaclavpe: <p>Hi,<br>
I am trying to compile PyTorch on older PC with NVidia GT 730. This card requires CUDA-11.4 as newer CUDAs no more support this card. CUDNN-8.2.4 matches to CUDA-11.4 therefore I am not able to use newer version of CUDNN.</p>
<p>How can I “downgrade” the PyTorch to be able to build with CUDA-11.4/CUDNN-8.2.4 setup ?</p>
<p>Thank you for help.</p>

ptrblck: <p>Try to <code>git checkout</code> an old release where Kepler GPUs (sm_35) were still supported.</p>

vaclavpe: <p>Hi, thank you for help. I somehow went through (actually switched off the CUDNN for a moment) but got the cmake error. Tried to google it but no success:</p>
<blockquote>
<p>CMake Error at cmake/public/cuda.cmake:262 (set_property):<br>
The link interface of target “torch::cufile” contains:</p>
<pre><code>CUDA::cuFile
</code></pre>
<p>but the target was not found.  Possible reasons include:</p>
<pre><code>* There is a typo in the target name.
* A find_package call is missing for an IMPORTED target.
* An ALIAS target is missing.
</code></pre>
<p>Call Stack (most recent call first):<br>
cmake/Dependencies.cmake:44 (include)<br>
CMakeLists.txt:865 (include)</p>
</blockquote>
<p>I was not able to find the CUDA::cuFile target anywhere. What am I missing? Thank you.</p>

ptrblck: <p>The PyTorch version you are trying to build might be too new for the CUDA toolkit you have installed (I don’t know which versions you are trying to build now). You could try to disable GDS via <code>USE_CUFILE=0</code> or could try my previous suggestion to downgrade PyTorch to an old version.</p>

vaclavpe: <p>Hi ptrblck,<br>
to be honest I gave up and bought newer NVidia card (Turing) and torch can work with CUDA now even without compilation. I was not able to find which older release to checkout and compile. I am sorry that I wasted your time. Just imagine how your father starts to install the Linux on his computer after 15 years of sitting behind Windows as a user. Funny picture, isn’t it ?</p>
Ensuring same predict results cross Python and C++

evilroach: <p>I have trained my model in Python, now I’m trying to export it to our producing env that must run model in c++.</p>
<p>I tried a few of ways(trace/script/onnx) to export the model, and basically got simimlar results.<br>
For example, I did things in Python as following:</p>
<pre><code>torch.set_grad_enabled( False )
model.eval()
model.freeze()
model.to_onnx( "path/to/file.onnx", input_sample=features, dynamo=True, export_params=True )
</code></pre>
<p>And then do inference in C++ according to the <a href="https://github.com/microsoft/onnxruntime-inference-examples/tree/main/c_cxx" rel="noopener nofollow ugc">ONNX Tutorials</a>.</p>
<p>The strange thing is that I got 43% results exactly same with the ones from Python.<br>
Because it’s a multi-task classification(6 label variables, and each var should be classified into 3 classes), I think that this ratio isn’t probabally a random results.</p>
<p>To avoid directly comaring float numbers, I didn’t comare the logits or probability values, instead apply an ArgMax on both of Python and C++ sides before I compare the integer class indexes.<br>
I’m a fresh man with DL, how can I trouble shot and where should I begin at?<br>
Thanks!!!</p>

Itamar_Katz: <p>It’s hard to impossible to make 2 different implementations output the same floating point number (bit-wise equality), and if one implementation outputs <code>[0.499, 0.501]</code> for a binary classification and the other <code>[0.501, 0.499]</code> (just as an example, it can be in the logits scale as well), you will get different output labels. So I would start by comparing the logits, and not the labels, to see if it is a matter of floating point different implementation, or something else.</p>

evilroach: <p>Thanks for your help!</p>
<p>I changed my comparing codes according to your advice,<br>
and almost all result probabilities matched when I give a very large tolerance(0.01) to <code>math.isclose()</code>.</p>
<p>To my supprise, the deviation of float point(FP32) computing is so large.</p>
Broken autograd momentum link

statgin: <p>The Towards Data Science link referencing <strong>momentum</strong> seems to be broken in the autograd tutorial (<a href="https://docs.pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">A Gentle Introduction to torch.autograd — PyTorch Tutorials 2.7.0+cu126 documentation</a>)</p>
<p>The article is instead linked to “<a href="https://towardsdatascience.com/solving-two-stage-stochastic-programs-in-gurobi-9372da1e3ba8/" rel="noopener nofollow ugc">Solving Two-Stage Stochastic Programs in Gurobi</a>”</p>

soulitzer: <p>Thanks for the report. That does seem to be the wrong link indeed. We’d be happy to accept a PR to fix this!</p>
JVP and checkpointing

Peter_Holderrieth: <p>I would like to compute a Jacobian vector product (JVP) with a large network. However, the transformer model uses checkpointing. If I compute the JVP, I get the error:<br>
<code>raise RuntimeError("Checkpointing is not compatible with .grad(), please use .backward() if possible")</code></p>
<p>Is this simple not possible then or are there solutions for this problem? I do not want to backpropagate through the output of the jvp (so I do not need checkpointing for that part of the code, e.g. if I could simply disable checkpointing for these lines of code, that would also be okay)</p>

soulitzer: <p>Do you have a stack trace of the error? You should only see that if you’re running backward.</p>
Oom error during process in 3d side project

reinforced: <p>My code link : <a href="https://github.com/genji970/gaussian-occupancy-prediction-with-nuscene" rel="noopener nofollow ugc">genji970/gaussian-occupancy-prediction-with-nuscene: using 3d gaussian splatting, generating dense voxel from sparse point cloud. Doing voxel occupancy prediction. Input is 6 multi view camera nuscene dataset and label is lidar nuscene dataset. pipeline from data process to traininig</a></p>
<pre><code class="lang-auto">======
Loading NuScenes tables for version v1.0-trainval...
23 category,
8 attribute,
4 visibility,
64386 instance,
12 sensor,
10200 calibrated_sensor,
2631083 ego_pose,
68 log,
850 scene,
34149 sample,
2631083 sample_data,
1166187 sample_annotation,
4 map,
Done loading in 68.971 seconds.
======
Reverse indexing ...
Done reverse indexing in 50.5 seconds.
======
  0%|          | 3/34149 [00:01&lt;5:01:35,  1.89it/s]
[DEBUG] Loaded 3 multiview samples
Traceback (most recent call last):
  File "...\train.py", line 132, in &lt;module&gt;
    occ_feat = occupancy_decoder(gaussian_embed, voxel_coords)
  File "...\.venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "...\gaussian_encoder\gaussian_decoder\gaussian_decoder.py", line 50, in forward
    dist = chunked_cdist(xyz, anchor_grid, chunk_size=512)  # (Nv, N)
  File "...\utils\ops.py", line 20, in chunked_cdist
    dist_chunk = torch.cdist(chunk, anchor_grid)  # (chunk_size, N)
  File "...\.venv\lib\site-packages\torch\functional.py", line 1222, in cdist
    return _VF.cdist(x1, x2, p, None)  # type: ignore[attr-defined]
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 MiB (GPU 0; 4.00 GiB total capacity; 6.88 GiB already allocated; 0 bytes free; 6.94 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
</code></pre>
<p>I did</p>
<pre><code class="lang-auto">1) reducing batch size
2) gc.collect() , torch.cuda.empty_cache()
3) with torch.cuda.amp.autocast(enabled=False):
4) os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:16"
5)chunking batch data 
</code></pre>
<p>But didn’t work</p>
<pre><code class="lang-auto">My env : rtx1650 4gib
</code></pre>
Multi-agent RL with different agent action spaces

acoursey: <p>I have been following the tutorial on multi-agent PPO from TorchRL: <a href="https://docs.pytorch.org/rl/stable/tutorials/multiagent_ppo.html" class="inline-onebox" rel="noopener nofollow ugc">Multi-Agent Reinforcement Learning (PPO) with TorchRL Tutorial — torchrl 0.8 documentation</a>. However, I am using the MPE2 speaker listener environment where both agents have different observation and action spaces.</p>
<p>Does TorchRL support this setting?</p>
<p>The MultiAgentMLP requires an int for the <strong>n_agent_outputs</strong>, not a list of ints. To get around this, I implemented my own network that supports different input and output shapes for each agent. However, when executing the ProbabilisticActor, the distribution only samples a single tensor, not one for each agent.</p>
<p>I would appreciate any advice on this topic!</p>
Using an AWS S3 bucket dataset with Torchvision.Datasets.ImageFolder

HugoH: <p>Hi all, I’m trying to load a large-scale dataset (imagery data) from an S3 bucket into my Sagemaker Jupyterlab instance to later start training it. However, when dealing with the data augmentations/variations using torchvisions datasets.ImageFolder there is a problem since ImageFolder does not accept lists nor S3MapDataset, it just accepts os.PathLike type.</p>
<p>I have been trying to address this problem using the S3 Plugin and the S3 connector for Pytorch but they load the data from S3 as other types rather than os.PathLike.</p>
<p>There have been similar questions on this platform: <a href="https://discuss.pytorch.org/t/can-i-use-torchvision-dataset-and-dataloader-with-aws-s3/34096" class="inline-onebox">Can I use torchvision Dataset and Dataloader with AWS S3?</a>, but seems there is still not a clear answer.</p>
<p>Is there any update as for June 2025 on this issue? Do you know any workaround that could solve this problem?</p>
What does cuda.is_initialized() actually check?

johann-petrak: <p>The documentation for the function <code>torch.cuda.is_initialized()</code> tells me</p>
<pre><code>"Returns whether PyTorch’s CUDA state has been initialized."
</code></pre>
<p>but what does that actually mean and when/for what do I need that information?</p>
<p>If I run this on a couple of different machines which support CUDA and where pytorch runs perfectly fine, I always get the return value <code>False</code>.</p>
<p>So: what exactly is this function telling me?</p>

ptrblck: <p><a href="https://github.com/pytorch/pytorch/blob/0dc99dcf59ce70145bec903ecabb5243ac2e4e2b/torch/cuda/__init__.py#L147-L149"><code>is_initialized()</code></a> checks if the internal <code>_initialized</code> flag was already set to <code>true</code> from <a href="https://github.com/pytorch/pytorch/blob/0dc99dcf59ce70145bec903ecabb5243ac2e4e2b/torch/cuda/__init__.py#L189-L235"><code>_lazy_init()</code></a> and also if the current process is in a “bad fork”, i.e. if you are trying to re-initialize CUDA in forked subprocess.</p>

yemaedahrav: <p>If it is in a bad fork, does it throw some error or just return False</p>
Pytorch data loading best practices, any good resources to explore?

Ptuser400: <p>Are there any good resources or tutorials for efficient and scalable data loading with pytorch? Please suggest.</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">This</a> is a great post describing common pitfalls and good practices.</p>
Evaluating gradients of output variables w.r.t parameters for pixelwise models

Joao-L-S-Almeida: <p>Hi.<br>
Consider I have a ViT  <code>NN</code> trained to perform segmentation which receives an input <code>x</code> of shape <code>(batch_size, n_channels, n_timesteps, height, width)</code> and outputs a prediction <code>y</code> of shape <code>(batch_size, n_classes, height, width</code>). Let us say that for each pixel <code>p</code> in the output image I want to evaluate the gradient with relation to all the model parameters <code>w</code> as <code>Grad(y_p, w) = \nabla_w NN(x; w)_p</code>, with final shape <code>(batch_size, n_classes, n_parameters)</code>.<br>
I would like to know what is the best approach to properly evaluate it using the frameworks available on PyTorch.</p>

valerian.rey: <p>Hi!<br>
Maybe the most straightforward way to do this is to iterate over the batch elements, over the classes and over the pixels, to get scalar tensors, from which you can compute the gradient with respect to the model’s parameters. I’m thinking of something like this:</p>
<pre data-code-wrap="python"><code class="lang-python">output: Tensor  # output.shape: [batch_size, n_classes, height, width]
for output_element in output:  # output_element.shape: [n_classes, height, width]
    for class_output in output_element:  # class_output.shape: [height, width]
        for y_row in class_output:  # y_row.shape: [width]
            for pixel in y_row:  # pixel.shape: [] (scalar)
                grads = torch.autograd.grad(pixel, model.parameters(), retain_graph=True)
                # grads will be a tuple with a many tensors as there are model parameters;
                # do whatever you need with grads
</code></pre>
<p>Obviously this will be very inefficient in terms of computational time, as it will require <code>batch_size * n_classes * height * width</code> backard passes through your model. If you also store the <code>grads</code>, it will require <code>batch_size * n_classes * height * width</code> times the size of your model in memory (which is most likely prohibitive, even with a small model).</p>
<p>There are ways to speed up the computation by parallelizing some of the backward passes using <code>torch.vmap</code>, or by using e.g. <a href="https://docs.pytorch.org/docs/stable/generated/torch.autograd.functional.jacobian.html" class="inline-onebox" rel="noopener nofollow ugc">torch.autograd.functional.jacobian — PyTorch 2.7 documentation</a>, but it will be way too memory intensive to compute all of the gradients in parallel. So the way you want to parallelize this computation depends on what you intend to do with your <code>grads</code>. If some of them could be summed together for instance, you could compute the result much more efficiently than what I showed.</p>
<p>Could you maybe provide more context?</p>

Joao-L-S-Almeida: <p>Hi, <a class="mention" href="/u/valerian.rey">@valerian.rey</a>, thank you the answer. I’m interested in using the metric seen here: <a href="https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&amp;arnumber=10759697" class="inline-onebox" rel="noopener nofollow ugc">IEEE Xplore Full-Text PDF:</a>, expression 4.</p>
RuntimeError: expected scalar type Float but found BFloat16

Federico_Ottomano: <p>I’m running my model in <code>bf16-mixed</code> precision and encountering the following error:</p>
<p><code>RuntimeError: expected scalar type Float but found BFloat16</code></p>
<p>This happens when I apply an nn.TransformerEncoder model to my data.</p>
<p>My input consists of two different modalities:</p>
<p><strong>Tensor H</strong>: Shape <code>(batch_size, 10000)</code>. I pass this through a 1D CNN feature extractor, then into a <code>nn.TransformerEncoder</code>. This entire pipeline works fine under <code>bf16-mixed precision</code>.</p>
<p><strong>Tensor C</strong>: Shape <code>(batch_size, 80)</code>. Here, I generate “tokens”, apply a padding mask, and then use an <code>nn.Embedding</code> layer before passing it to the transformer. This is where I hit the error mentioned above.</p>
<p>Since everything works for <strong>Tensor H</strong>, I suspect this issue may be specific to how <code>nn.Embedding</code> behaves under <code>bf16-mixed</code> precision.</p>
<p>Below the code:</p>
<pre><code class="lang-auto"># ... other stuff above..

    if self.spectrum_type == 'cnmr' and self.cnmr_binary: # This is Tensor A
        # Tokenize the binary CNMR data

        tokens = self._tokenize_cnmr(x)
        
        # Create mask for transformer (True values will be ignored)
        mask = (tokens == 0).bool()
        
        # Embed the tokens (this is our feature extraction)
        x = self.feature_extractor(tokens)  # [batch_size, seq_len, d_model
        
        # Apply transformer with mask
        x = self.transformer(x, src_key_padding_mask=mask) # This throws the error :'(

        return x, mask
        
    else:  # Below Tensor H
        # Add channel dimension
        x = x.unsqueeze(1)  # [batch_size, 1, sequence_length]
        
        # Apply integrated feature extraction (including pooling if configured)
        x = self.feature_extractor(x)  # [batch_size, channels, seq_len]
        
        # Reshape for projection
        x = x.transpose(1, 2)  # [batch_size, seq_len, channels]
        x = self.post_conv_proj(x)  # [batch_size, seq_len, d_model]
        
        # Add positional encoding if needed
        if self.use_pos_encoding:
            x = self.pos_encoding(x)
        
        # Apply transformer encoder
        x = self.transformer(x)
        
        # Create empty mask (no padding)
        mask = torch.zeros(x.shape[0], x.shape[1], dtype=torch.bool, device=x.device)
        
        return x, mask

# ... other stuff below...
</code></pre>
<p>Any guidance on how to resolve this, or whether explicit casting is needed somewhere, would be appreciated!</p>

ptrblck: <p>Are you using <code>autocast</code> or do you manually cast the data?</p>
Preserving bits like conj, neg during functionalization

cz2h: <p>Hi team, I am looking into <code>__torch_dispatch__</code> under <code>functional_tensor.py</code>.</p>
<p>During the forward pass there is logic of cloning arg tensor and call the decomposition function, but the bits like conj/neg are resolved during the clone process, this might differ from the real usage where the real input can contain these bits in it.</p>
<p>Are there any concerns that vioalte the design if we some how preserve these bits?</p>
Cannot build PyTorch 2.3.1 with CUDA 12.9 on WSL2 (Ubuntu 22.04) — Missing nvToolsExt despite cuda-nvtx-12-9 installed

sabi-ant: <p>Hi all,</p>
<p>I’m trying to build <strong>PyTorch v2.3.1 from source</strong> on a <strong>WSL2 Ubuntu 22.04 environment</strong> with the following setup:<br>
Any way to build pytorch 2.3.1 with cuda 12.9?<br>
Cause it seems latest pytorch incompatible with some modules(TI QAT wrapper module) I need to use in my project, I need pytorch 2.3.1.</p>
<hr>
<h3><a name="p-471666-system-configuration-1" class="anchor" href="#p-471666-system-configuration-1"></a><img src="https://discuss.pytorch.org/images/emoji/apple/white_check_mark.png?v=14" title=":white_check_mark:" class="emoji" alt=":white_check_mark:" loading="lazy" width="20" height="20"> System Configuration:</h3>
<ul>
<li><strong>OS:</strong> WSL2 (Ubuntu 22.04)</li>
<li><strong>GPU:</strong> NVIDIA RTX 5080</li>
<li><strong>CUDA Driver Version:</strong> 12.9</li>
<li><strong>CUDA Toolkit Version:</strong> 12.9 (installed via APT)</li>
<li><strong>cuDNN:</strong> libcudnn9 installed via APT</li>
<li><strong>MAGMA:</strong> <code>magma-cuda121</code> installed via conda</li>
<li><strong>Python:</strong> Conda environment, Python 3.10</li>
<li><strong>GCC:</strong> 11.4</li>
</ul>
<p>Environment variables:</p>
<pre><code class="lang-auto">export CUDA_HOME=/usr/local/cuda-12.9
export _GLIBCXX_USE_CXX11_ABI=1
export USE_CUDA=1
export USE_CUDNN=1
export USE_MKLDNN=1
export USE_MAGMA=1
export USE_NCCL=0
export MAX_JOBS=$(nproc)
export _GLIBCXX_USE_CXX11_ABI=1
export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
</code></pre>
<hr>
<h3><a name="p-471666-problem-2" class="anchor" href="#p-471666-problem-2"></a><img src="https://discuss.pytorch.org/images/emoji/apple/warning.png?v=14" title=":warning:" class="emoji" alt=":warning:" loading="lazy" width="20" height="20"> Problem:</h3>
<p>While building PyTorch 2.3.1 using <code>python setup.py develop</code>, the process fails with the following error:</p>
<pre><code class="lang-auto">CMake Error at cmake/public/cuda.cmake:70 (message):
  Failed to find nvToolsExt
</code></pre>

ptrblck: <aside class="quote no-group" data-username="sabi-ant" data-post="1" data-topic="220728">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sabi-ant/48/77333_2.png" class="avatar"> sabi-ant:</div>
<blockquote>
<p>Any way to build pytorch 2.3.1 with cuda 12.9?</p>
</blockquote>
</aside>
<p>No, I would not recommend trying to build old PyTorch releases with newer CUDA toolkits as needed changes might not have landed as can be seen in your error message.</p>

sabi-ant: <p>Ok, it is very clear.</p>
Simple "hello world" with torchtune on mac

braingal: <p>Hi, is it possible to run a simple lora fine tuning job on my mac, no gpu, with torchtune.<br>
I’ve followed the <a href="https://docs.pytorch.org/torchtune/stable/tutorials/first_finetune_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Fine-Tune Your First LLM — torchtune 0.6 documentation</a> but I run out of memory and there’s possibly a hard requirement for a GPU?<br>
Is there a recipe that can be used just to try this out locally on my mac?<br>
Thanks</p>

roach: <p>Macs use ‘metal plate shaders’ or ‘mps’ and unified memory, so unless your total system memory is shot you should be okay. Don’t forget to set the device to ‘mps’ instead of ‘cpu’ and of course don’t use ‘cuda’. There are some issues with non mps implemented things, so you should set a system var (export PYTORCH_ENABLE_MPS_FALLBACK=1) to enable fall back to CPU, you probably will get the relevant errors that point that out. If it’s just memory related, tough luck, you have to either reduce the inputs, or use a less demanding module.</p>