akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Issue with Ground Truth Mapping: Class IDs from a Different Dataset Mapped to COCO Class Names

username: AGH
cooked: <p>Im using a pretrained model(yolov8 (ultralytics)) which is trained on COCO dataset.(COCO has 80 classes). Now I am validating it on medical-pills dataset which has only 1 class.</p>
<p>After running this code,</p>
<p>from ultralytics import YOLO</p>
<h1><a name="p-467730-load-yolov8-pose-model-1" class="anchor" href="#p-467730-load-yolov8-pose-model-1"></a>Load YOLOv8 pose model</h1>
<p>model = YOLO(“yolov8n.pt”)</p>
<h1><a name="p-467730-validate-on-the-hand-keypoints-dataset-2" class="anchor" href="#p-467730-validate-on-the-hand-keypoints-dataset-2"></a>Validate on the hand keypoints dataset</h1>
<p>metrics = model.val(data=“/content/medical-pills.yaml”, save_json=True)</p>
<h1><a name="p-467730-print-results-3" class="anchor" href="#p-467730-print-results-3"></a>Print results</h1>
<p>print(metrics)</p>
<p>Im getting all evaluation metrics and also val_batch0_labels.jpg and val_batch0_pred.jpg</p>
<p>In val_batch0_labels.jpg i am getting pills images all with bounding boxes detected as person(which is the first class in COCO)</p>
<p>and in val_batch0_pred.jpg<br>
i am getting pills images as cup,cake,dining table,bottle(which all are classes of COCO dataset)</p>
<p>Why its taking pill as person,instead of taking it as pill.</p>
<p>Is there any class mismatch? Or is the code taking yaml file of COCO?</p>
Dataset handling and model selection

username: Muhammad_Irfan
cooked: <p>I am facing a problem, i have a point error dataset of 74 rows in which 3 are my inputs and 16 are output features. Well when I trained my model using polynomial regression, if I increased the degree I got good results in train r2 and test r2 and this is overfitting. when I used DNN model with complex hidden layer I got good train r2 but test r2 fails so what you suggest to get good train and test r2, either I need to change my model as when I tuned my polynomial model the results not improved or we have to collect more data ?</p>
Torch.xpu.synchronize() hangs, 2.6.0 on intel core ultra 185h

username: KFrank
cooked: <p>Hi All!</p>
<p><code>torch.xpu.synchronize()</code> hangs on pytorch 2.6.0+xpu.</p>
<p>Steps to reproduce:</p>
<pre><code class="lang-plaintext">(2_6_0_xpu) $ python
Python 3.12.9 | packaged by Anaconda, Inc. | (main, Feb  6 2025, 18:56:27) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.6.0+xpu'
&gt;&gt;&gt; t = torch.randn (10000, 10000, device = 'xpu')
&gt;&gt;&gt; for i in range (100):
...     u = t @ t
... 
&gt;&gt;&gt; torch.xpu.synchronize()
Terminated
(2_6_0_xpu) $ 
</code></pre>
<p>Note, the loop over <code>u = t @ t</code> takes about 30 seconds to run (but the <code>for</code> command<br>
returns promptly), while <code>torch.xpu.synchronize()</code> is launched within a second or two<br>
of starting the <code>for</code> loop.</p>
<p>In this circumstance, <code>torch.xpu.synchronize()</code> hangs apparently indefinitely.  (I let it run<br>
for five minutes before killing the python process from a separate terminal window.)</p>
<p>During this time <code>intel_gpu_top</code> shows, after about thirty seconds, that the xpu is no longer<br>
in use, while top (and the ubuntu system monitor) show the python process consuming<br>
“100%”  of the cpu.</p>
<p>If <code>torch.xpu.synchronize()</code> is launched after the <code>u = t @ t</code> loop has actually completed,<br>
it doesn’t hang, and it doesn’t seem to hang if it is launched while a shorter, three-second<br>
<code>for i in range (10):</code> loop is still running.</p>
<p>Pytorch with xpu support was installed in a newly-created conda environment following<br>
<a href="https://pytorch.org/docs/main/notes/get_start_xpu.html" rel="noopener nofollow ugc">this pytorch documentation</a> using:</p>
<pre><code class="lang-plaintext">pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/xpu
</code></pre>
<p>(I had to copy /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.33 into the conda environment,<br>
replacing the conda environment’s libstdc++.so.6.0.29, in order to get the xpu to work,<br>
although the details of this are confused.)</p>
<p>This is on ubuntu 24.04.2 LTS with an intel core ultra 185h processor (and also an<br>
nvidia graphics chip).</p>
<p>Thanks for any information or updates on the status of 2.6.0+xpu.</p>
<p>K. Frank</p>
Batch size at inference is influencing accuracy

username: danbull-scanabull
cooked: <p>Ive created a custom computer vision model which randomly samples points from a LiDAR point cloud, and performs semantic segmentation. This is a fairly simple model, essentially embedding the raw points, performing batch normalisation, and running through a series of 8 transformer blocks. A set of dense layers in the head creates an output, which separates the points in to two classes. This uses a crossentropy loss.<br>
At training model maxes out at about 97.5% accuracy. However my results at test depend on batch size i.e.:<br>
batchsize = 1: accuracy = 91%<br>
batchsize = 2: accuracy = 95%<br>
batchsize = 3: accuracy = 96%<br>
batchsize =4: accuracy = 97%<br>
batchsize 20: accuracy = 97.4%<br>
Why should this be the case?<br>
Im using model.eval(), torch.no_grad(). Ive this with/without batchnormalisation - no massive difference.</p>
Pip3 install torch locked on 2.2.2

username: DSquid
cooked: <p>I am on an old intel chip mac (in case that is the problem)…  and for the life of me, I can’t get anything later than 2.2.2 version of torch from pip.  I promise I have read the release notes starting at 2.3…  but it doesn’t mention anything about dropping mac/intel.  Anyone have a clue?</p>

username: ptrblck
cooked: <p>You are right and PyTorch 2.2.x was the last release supporting macOS x86 builds as also mentioned <a href="https://discuss.pytorch.org/t/why-no-macosx-x86-64-build-after-torch-2-2-2-cp39-none-macosx-10-9-x86-64-whl/204546/2">here</a>.</p>
Libtorch multi GPU training

username: kevin2004
cooked: <p>Hi guys, I am wondering if multiple GPU training (libtorch) is supported in 2025.</p>
<p>See this link: <a href="https://github.com/pytorch/pytorch/issues/48959" class="inline-onebox" rel="noopener nofollow ugc">[Poll] Support DistributedDataParallel (DDP) in PyTorch C++ API (libtorch) · Issue #48959 · pytorch/pytorch · GitHub</a></p>

username: kevin2004
cooked: <p>I am stilling waiting for the answers. Thanks.</p>
Training Machine Learning Model In Browser For Reinforcement Learning

username: Sliferslacker
cooked: <p>I’m trying to make a game website and add some reinforcement learning into the games. So my backend for making the website is Django, and I’m currently using PyTorch for the neural network. I’ve made the game on the web browser where the logic of the game is inside a javascript file. I’m wondering how would I go about importing the untrained PyTorch neural network to the browser to be trained?</p>
<p>Because if I wanna train the model in the backend, then I would have to rewrite the whole game logic in python just to train it then import it. Since I’ve already written the game in javascript file and imported it to the browser, I would like import neural network to browser and then train it. Would this be easier if I switched to Tensorflow and use Tensorflow.js?</p>
torch.cuda.is_available()=False;torch._C._cuda_getDeviceCount() > 0

username: Kid
cooked: <p>I used the official mirror from the PyTorch website:</p>
<pre><code class="lang-auto">docker pull pytorch/pytorch:2.5.1-cuda12.1-cudnn9-devel
</code></pre>
<p>My host machines are WSL2 and CentOS8 (I tried both, and they produced the same error)，and their cuda driver are cuda 12.4,  (one is 4090, another is Nvidia A10).</p>
<p>The <code>nvidia-smi</code> output is normal, but <code>torch.cuda.is_available()</code> returns <code>False</code>. However, when I run:</p>
<pre><code class="lang-auto">CUDA_DEVICE_ORDER="PCI_BUS_ID" PYTORCH_NVML_BASED_CUDA_CHECK=1 python -c "import torch; print(torch.cuda.is_available())"
</code></pre>
<p>It returns <code>True</code>.</p>
<p>But in practice, when I run:</p>
<pre><code class="lang-auto">CUDA_DEVICE_ORDER="PCI_BUS_ID" PYTORCH_NVML_BASED_CUDA_CHECK=1 python demo.py
</code></pre>
<p>It still throws the following error (where <code>demo.py</code> is my LLM inference script).</p>
<p>I’ve also tried other official versions:</p>
<ul>
<li><code>pytorch/pytorch:2.4.0-cuda12.4-cudnn9-devel</code></li>
<li><code>pytorch/pytorch:2.3.1-cuda12.4-cudnn8-devel</code></li>
</ul>
<p>But <code>torch.cuda.is_available()</code> still returns <code>False</code>.</p>
<p>Surprisingly, I have another image (<code>pytorch:2.3.1-cuda12.4-cudnn8</code>), which was <strong>not</strong> pulled from the official PyTorch repository, and it works perfectly fine.</p>
<p>Help!! Please save me!! <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=12" title=":sob:" class="emoji" alt=":sob:" loading="lazy" width="20" height="20"></p>
<p><strong>Why? Why? Why?</strong></p>
<pre><code class="lang-auto">&gt;&gt;&gt; torch.cuda.device_count()
6
&gt;&gt;&gt; torch.cuda.is_available()
/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py:129: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 304: OS call failed or operation not supported on this OS (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() &gt; 0
False
&gt;&gt;&gt; torch.cuda.current_device() 
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 940, in current_device
    _lazy_init()
  File "/usr/local/lib/python3.10/dist-packages/torch/cuda/__init__.py", line 319, in _lazy_init
    torch._C._cuda_init()
RuntimeError: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 304: OS call failed or operation not supported on this OS
&gt;&gt;&gt; torch.cuda.device(0)
&lt;torch.cuda.device object at 0x7f697c8f4070&gt;
</code></pre>
How does PyTorch's cross-entropy loss transform logits with a soft probability target vector?

username: N_Beladel
cooked: <p>If I understand correctly, PyTorch’s <code>nn.CrossEntropyLoss()</code> accepts soft probability target vectors. So, for example, y (the target vector) could be [0.8, 0.2, 0.8]^T, representing an equal (0.8) probability for classes 1 and 3, and a probability of 0.2 for the second class. Obviously, this is not a normalized probability distribution, as it sums to 1.2. Now, x are the raw logits produced by the model; does <code>nn.CrossEntropyLoss()</code> still apply a regular (log)softmax to the logits? If so, as I understand it, the logits are transformed to a normalized probability distribution, and will never match the un-normalized soft probability target. If the (log)softmax is not applied, how does the loss function act on the raw logits instead (if at all)? Thank you in advance!</p>

username: KFrank
cooked: <p>Hi Nassim!</p>
<aside class="quote no-group" data-username="N_Beladel" data-post="1" data-topic="218037">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/n_beladel/48/76250_2.png" class="avatar"> N_Beladel:</div>
<blockquote>
<p>this is not a normalized probability distribution, as it sums to 1.2. Now, x are the raw logits produced by the model; does <code>nn.CrossEntropyLoss()</code> still apply a regular (log)softmax to the logits?</p>
</blockquote>
</aside>
<p>Yes, the <code>input</code> consists of unnormalized log-probabilities (contrary to the documentation,<br>
they’re not really logits) and are treated the same whether or not the “probabilities” that<br>
make up the soft <code>target</code> form a properly-normalized probability distribution.  (The values<br>
in <code>target</code> don’t even have to be individually proper probabilities between zero and one.<br>
They just get plugged into the cross-entropy formula without being checked or normalized.)</p>
<p>The documentation is silent (as far as I can tell) on this issue, but a quick test shows it to<br>
be the case:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'2.6.0+cu126'
&gt;&gt;&gt; _ = torch.manual_seed (2025)
&gt;&gt;&gt; l = torch.randn (10)
&gt;&gt;&gt; t = torch.rand (10)
&gt;&gt;&gt; torch.nn.CrossEntropyLoss() (l, t)
tensor(15.4764)
&gt;&gt;&gt; torch.nn.CrossEntropyLoss() (l, 10 * t)
tensor(154.7643)
&gt;&gt;&gt; torch.nn.CrossEntropyLoss() (l, 0.1 * t)
tensor(1.5476)
&gt;&gt;&gt; torch.nn.CrossEntropyLoss() (l, -t)
tensor(-15.4764)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Data loading time becomes unstable (jumping to high values) when dataset is large

username: Sihe_Chen
cooked: <p>I am using a dataset that is quite large (like 5 Terabytes).<br>
In testing, I used small datasets, using like 300 GB of it; by doing this, the dataloading time is as short as 0.1 seconds. However, when I use the full dataset, the time starts to become variable, going from 0.1 to around 1.0 seconds. Here are two approaches that I have tried:</p>
<ol>
<li>Unified the input to a 2D tensor, combined them into a large binary file, and directly read from a random location of the large binary file</li>
<li>Pickle the data into a LMDB file, and take the data from a random key.</li>
</ol>
<p>Both of the approaches work with smaller datasets but becomes worse with large dataset. Any ideas how this can be happening and how we can solve the problem?</p>
<p>P.S. <em>I am using DDP for training, so one way that I could think of is to shatter the whole dataset to 8, and since I have 8 GPUs I can just make each card read from each database. Suggestions about this idea would be welcome too!</em></p>
Will a tensor still be in pinned memory which is stacked by two pinned memory

username: Igo312
cooked: <p>Here is the example, I want to ask will the <code>tensor_res</code> still be in pinned memory?</p>
<pre><code class="lang-auto">tensor_a = torch.zeros([1, 3]).pin_memory()
tensor_b = torch.zeros([1, 3]).pin_memory()
tensor_res = torch.cat([tensor_a, tensor_b], dim=0)
</code></pre>

username: ptrblck
cooked: <p>No, it won’t:</p>
<pre data-code-wrap="python"><code class="lang-python">&gt;&gt;&gt; tensor_a.is_pinned()
True
&gt;&gt;&gt; tensor_b.is_pinned()
True
&gt;&gt;&gt; tensor_res.is_pinned()
False
</code></pre>
How to store temp variables with register_autograd without returning them as output?

username: beschtaPy
cooked: <p>Hi!</p>
<p>I’d like to write a custom op for a c++ operation to enable autograd and compile support.<br>
However, my c++ function returns two tensors: 1) the output and 2) temporary values only needed for the backward pass.</p>
<p>When inheriting from torch.autograd.function, you can easily “wrap” the c++ function, store the temp values in ctx and return only the single output tensor. How can I do the same with register_autograd?</p>
<p>E.g.</p>
<pre><code class="lang-auto">
@torch.library.custom_op("custom_fun", mutates_args=())
def custom_fun(x):
   out, tmp = _C.run(x)
   return out, tmp # this returns 2 outputs instead of only 'out'

def setup_context(ctx, input, output):
   out, tmp = output # I need to get tmp here somehow for backward, but don't want to have it as output
   ctx.tmp = tmp
   ctx.save_for_backward(input)

@torch.library.custom_op("backward_impl", mutates_args=())
def backward_impl(x, dx, tmp):
    return _C.run_back(x, dx, tmp)

def backward_fun(ctx, dx):
   tmp = ctx.tmp
   x = ctx.saved_tensors
   return backward_impl(x, dx, tmp)

custom_fun.register_autograd(backward_fun, setup_context=setup_context)

out, tmp = custom_fun(data) # this returns 2 values!

</code></pre>

username: FSchoen
cooked: <p>Hi! Have you since found a solution for this?</p>
<p>The only thing I can come up with at the moment is to wrap <code>custom_fun</code> in a wrapper function which simply discards the second output needed for the backwards pass, but this seems a bit convoluted.</p>
Restrict FCN output to valid convolutions

username: cpegel
cooked: <p>Hello,</p>
<p>I’m using <a href="https://github.com/qubvel-org/segmentation_models.pytorch" rel="noopener nofollow ugc">Segmentation Models PyTorch</a> to create and train a FCN for a semantic segmentation task. For training, I chop up my training data into patches of the same size (512×512). Then I am applying the model to larger inputs for inference, which is possible for FCN architectures using PyTorchs dynamic computional graphs. I just have to make sure the input dimensions are a multiple of the FCNs strides, in my case 32 pixels, which can easily be achieved by padding the inputs.</p>
<p>However, I now want to perform the inference on hardware with rather limited GPU memory, so I can’t feed the large inputs directly into the network. Instead I have to chop it into patches and put them back together. In order for this to produce the same results as when the network is fed the whole image, I have to restrict the output of the network to only those pixels, where the involved convolutions don’t see any of the network-internal padding, so only the “valid convolutions”. In other words, only those output pixels, whose receptive fields don’t see beyond the given image patch.</p>
<p>Is it possible to restrict the output of a FCN in PyTorch to those pixels, where all convolutions are valid in the sense that they don’t depend on padding?</p>
5070Ti+Ubuntu 20.04.6+cuda?

username: riva_lei
cooked: <p>I want to use 5070ti to run my dl model. Which cuda and pytorch versions are compatible?</p>

username: ptrblck
cooked: <p>The nightly PyTorch binaries with CUDA 12.8 are compatible with Blackwell GPUs.</p>

username: riva_lei
cooked: <p>Get it! I will try it,Thanks!</p>
CNN loss is not decreasing

username: Saqlain_Afroz
cooked: <p>I am training a CNN model (for which I tried with both, my custom CNN and VGG19 model) with a dataset with 3 classes. Starting from 1st epoch the loss is staying almost constant. I will attach my colab’s link below for your reference: <a href="https://colab.research.google.com/drive/1dHi9wHWuBuLOrOgWCtmssRT6_GaLwMMa?usp=sharing" rel="noopener nofollow ugc">Colab Link</a></p>
<p>My training losses for the 1st few epochs looks like this:</p>
<blockquote>
<p>Epoch: 0 | Train loss: 1.4142, Train accuracy: 0.3319 | Test loss: 1.1003, Test accuracy: 0.3333</p>
<p>Epoch: 1 | Train loss: 1.1002, Train accuracy: 0.3369 | Test loss: 1.1008, Test accuracy: 0.3333</p>
<p>Epoch: 2 | Train loss: 1.1006, Train accuracy: 0.3344 | Test loss: 1.0991, Test accuracy: 0.3333</p>
<p>Epoch: 3 | Train loss: 1.1006, Train accuracy: 0.3313 | Test loss: 1.0990, Test accuracy: 0.3333</p>
</blockquote>
<p>I have tried this same approach for longer epochs but it doesn’t converge. So, for like 50 epochs also the loss and accuracy remains the same.</p>
<p>Even if the loss is reducing, it’s very slow.</p>
<p>I have used the same model on different dataset, and for that also the loss and accuracy remains same for like 100 epochs.</p>
Simulating quantization to lower bit precision with quant_min/max setting on fused modules

username: TominoFTW
cooked: <p>Hi,<br>
I am experimenting with simulating of different bit levels of quantization for my QAT on ResNet18.<br>
I set up a custom qconfig for selected modules when initializing the model e.g.:</p>
<pre><code class="lang-auto">                self.qconfig = quant.get_default_qat_qconfig("x86")
                ...
                # activation_bits and weight_bits loaded from config file
                custom_qconfig = quant.QConfig(
                    activation = quant.MinMaxObserver.with_args(
                        quant_min=0,
                        quant_max=(2**activation_bits - 1) // 2,
                        dtype=torch.quint8,
                        qscheme=torch.per_tensor_affine,
                    ),
                    weight=weight_observer.with_args(
                        quant_min=-((2 ** (weight_bits - 1))) // 2,
                        quant_max=(2 ** (weight_bits - 1) - 1) // 2,
                        dtype=torch.qint8,
                        qscheme=weight_scheme,
                    ),
                )
</code></pre>
<p>Then I follow the steps from QAT tutorial, hence running <code>fuse_modules_qat</code> and <code>prepare_qat</code> functions. But after this step it feels like that my changes to quant_min/max are not being propagated into the fused module with whatever changes I do to min/max args. Maybe they are not supposed to be, since those modules are getting the attributes from the default <code>self.qconfig = quant.get_default_qat_qconfig("x86")</code>.</p>
<pre><code class="lang-auto"># output of print(model.conv1)

ConvBnReLU2d(
  3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False
  (bn): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (weight_fake_quant): FusedMovingAvgObsFakeQuantize(
    fake_quant_enabled=tensor([1], device='cuda:0'), observer_enabled=tensor([1], device='cuda:0'), scale=tensor([1.], device='cuda:0'), zero_point=tensor([0], device='cuda:0', dtype=torch.int32), dtype=torch.qint8, quant_min=-128, quant_max=127, qscheme=torch.per_channel_symmetric, reduce_range=False
    (activation_post_process): MovingAveragePerChannelMinMaxObserver(min_val=tensor([], device='cuda:0'), max_val=tensor([], device='cuda:0'))
  )
  (activation_post_process): FusedMovingAvgObsFakeQuantize(
    fake_quant_enabled=tensor([1], device='cuda:0'), observer_enabled=tensor([1], device='cuda:0'), scale=tensor([1.], device='cuda:0'), zero_point=tensor([0], device='cuda:0', dtype=torch.int32), dtype=torch.quint8, quant_min=0, quant_max=127, qscheme=torch.per_tensor_affine, reduce_range=True
    (activation_post_process): MovingAverageMinMaxObserver(min_val=inf, max_val=-inf)
  )
)
</code></pre>
<p>My question is, if there is any way to do that, since I would like to have different levels of “quantization” on each layer.<br>
Maybe I am just missing something as I am still new into this so any help is appreciated!</p>
<p>Thanks in advance!</p>
Errors while converting Def-DETR pth to .onnx

username: Sarvesh_Shashikumar
cooked: <p>I am attempting convert a Def-DETR model to onnx.</p>
<p>batch_dim = Dim(“batch_size”, min=1, max=4)<br>
images_dim = Dim(“dimensions”)<br>
images_height = Dim(“height”)<br>
images_width = Dim(“width”)</p>
<pre><code>torch.onnx.export(model,         # model being run 
    model_inputs,       # model input (or a tuple for multiple inputs) 
    "model.onnx",       # where to save the model  
    export_params=True,  # store the trained parameter weights inside the model file 
    opset_version=16,    # the ONNX version to export the model to 
    input_names=['images', 'masks'],   # the model's input names 
    output_names = ['out'], # the model's output names 
    dynamo=True,
    dynamic_shapes={
        "images": {0: batch_dim, 1: images_dim, 2: images_height, 3: images_width},
        "masks": {0: batch_dim, 1: images_height, 2: images_width}
    },
    # report=True,
    # verify=True,
    verbose=True
)
</code></pre>
<p>I am facing the below error.</p>
<h2><a name="p-467574-exception-summary-1" class="anchor" href="#p-467574-exception-summary-1"></a>Exception summary</h2>
<p>&lt;class ‘torch.fx.experimental.symbolic_shapes.GuardOnDataDependentSymNode’&gt;: Could not extract specialized integer from data-dependent expression u0 (unhinted: u0).  (Size-like symbols: none)</p>
<p>Caused by: (_export/non_strict_utils.py:557 in <strong>torch_function</strong>)<br>
For more information, run with TORCH_LOGS=“dynamic”<br>
For extended logs when we create symbols, also add TORCHDYNAMO_EXTENDED_DEBUG_CREATE_SYMBOL=“u0”<br>
If you suspect the guard was triggered from C++, add TORCHDYNAMO_EXTENDED_DEBUG_CPP=1<br>
For more debugging help, see <a href="https://docs.google.com/document/d/1HSuTTVvYH1pTew89Rtpeu84Ht3nQEFTYhAX3Ypa_xJs/edit?usp=sharing" class="inline-onebox" rel="noopener nofollow ugc">Dealing with GuardOnDataDependentSymNode errors - Google Docs</a></p>
<p>For C++ stack trace, run with TORCHDYNAMO_EXTENDED_DEBUG_CPP=1</p>
<p>The following call raised this error:<br>
File “models/deformable_transformer.py”, line 241, in get_reference_points<br>
ref_y, ref_x = torch.meshgrid(torch.linspace(0.5, H_ - 0.5, H_, dtype=torch.float32, device=device),</p>
<p>(Refer to the full stack trace above for more information.)</p>
<p>Would be grateful if i get a solution. Thanks in advance!</p>
Fuse adds in TorchScript

username: kamei
cooked: <p>Is there a method in a graph that allows me to specify where a constant will be placed in the code?<br>
Here’s an example:</p>
<p><strong>original graph</strong><br>
%3 : int = prim::Constant value=1()<br>
%2 : int = prim::Constant value=2()<br>
%5 : int = prim::Constant value=4()<br>
%9 : int = prim::Constant value=3()<br>
%4 : Tensor = aten::add(%x.1, %2, %3)<br>
%a.1 : Tensor = aten::add(%4, %5, %3)<br>
%b.1 : Tensor = aten::add(%a.1, %9, %3)<br>
%c.1 : Tensor = aten::relu(%b.1)<br>
return (%c.1)</p>
<hr>
<p><strong>fuse_ops function</strong><br>
value1 = current_input2.toIValue()<br>
value2 = next_input2.toIValue()<br>
<strong>--------------------insertConstant------------------------</strong><br>
new_int = graph.insertConstant(value1 + value2)</p>
<p><strong>--------------------------create-----------------------------</strong><br>
new_int_2 = graph.create(‘prim::Constant’)<br>
new_int_2.f_( “value”, value1 + value2)<br>
new_int_2.insertBefore(nodes[0])<br>
new_add = graph.create(“aten::add”, [current_input1, new_int, current_alpha])<br>
new_add.insertBefore(current)<br>
next_node.output().replaceAllUsesWith(new_add.output())<br>
<strong>---------------------------------------------------------------</strong></p>
<hr>
<p><strong>fuse_ops graph</strong><br>
graph(%x.1 : Tensor):<br>
%25 : Tensor = prim::Constant value=7. () # graph.create<br>
%3 : int = prim::Constant value=1 ()<br>
%2 : int = prim::Constant value=2 ()<br>
%5 : int = prim::Constant value=4 ()<br>
%9 : int = prim::Constant value=3 ()<br>
%4 : Tensor = aten::add(%x.1, %2, %3)<br>
%26 : Tensor = aten::add(%4, %24, %3)<br>
%c.1 : Tensor = aten::relu(%26)<br>
%24 : int = prim::Constant value=7 () # insertConstant<br>
return (%c.1)</p>
<hr>
<p>I tried two approaches: using <code>graph.insertConstant</code> and <code>graph.create</code>. Neither was successful.</p>
<ul>
<li><code>graph.insertConstant</code> only inserts the constant at the bottom of the code (as seen in the <code>fuse_ops</code> graph).</li>
<li><code>graph.create</code> creates a node, which causes an issue when generating an <code>aten::add</code>.</li>
</ul>
Multiple Models Performance Degrades

username: goofydoge
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c795da9ca2096fc66e9c4b0b865a7995bda649b.png" data-download-href="https://discuss.pytorch.org/uploads/default/1c795da9ca2096fc66e9c4b0b865a7995bda649b" title="chrome_K0u4KDMccQ"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c795da9ca2096fc66e9c4b0b865a7995bda649b_2_690x212.png" alt="chrome_K0u4KDMccQ" data-base62-sha1="43TorTDSIX4BRzl9GWauUgmSJOr" width="690" height="212" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c795da9ca2096fc66e9c4b0b865a7995bda649b_2_690x212.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c795da9ca2096fc66e9c4b0b865a7995bda649b_2_1035x318.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c795da9ca2096fc66e9c4b0b865a7995bda649b_2_1380x424.png 2x" data-dominant-color="383737"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">chrome_K0u4KDMccQ</span><span class="informations">1389×428 87.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Hello all, I have a custom Lightning implementation where I use MONAI’s UNet model for 2D/3D segmentation tasks. Occasionally while I am running training, every model’s performance drops drastically at the same time. I’m hoping someone can point me in the right direction on what could cause this.</p>
<p>I run a baseline pass with basic settings and no augmentations (the grey line). I then make adjustments (different ROI size, different loss function, etc.). I then start training a model on GPU 0 with variations from the baseline, and I repeat this for the amount of GPUs that I have. So I have GPU 1 with another model variation running, GPU 2 runs another model variation, etc. I have access to 8x GPU, and I generally do this in order to speed up the process of finding a good model. (I’m a novice so there’s probably a better way to do this, too)</p>
<p>All the models access the same dataset. Nothing is changed in the dataset.</p>
Testing .pte model with all ones give different result on PC and Android

username: Sylte
cooked: <p><strong>Edit:</strong> Admins please delete this thread. I found out I was using an old model file on the handset. The entire thing is user error.</p>
<p>I’m very sorry<br>
/Sylte</p>

username: Sylte
cooked: <p>My mistake, this should not have been posted in the first place.</p>

username: Sylte
cooked: <p>My mistake, this should not have been posted.</p>
Use subset of dataset in c++ frontend

username: Wousta
cooked: <p>I am using MNIST in the c++ frontend, and I want to use a reduced version of it. However, the c++ frontend does not appear to have torch.utils.data.Subset or RandomizedSubsetSampler, how would I do it then?</p>
<p>I am using the MNIST example from <a href="https://github.com/pytorch/examples/blob/main/cpp/mnist/mnist.cpp" class="inline-onebox" rel="noopener nofollow ugc">examples/cpp/mnist/mnist.cpp at main · pytorch/examples · GitHub</a></p>
<p>Thanks!</p>
How to Export `torchaudio.models.decoder.ctc_decoder` to TorchScript for C++ Deployment?

username: mariaalfaroc
cooked: <p>Hello everyone,</p>
<p>I’m trying to export <code>torchaudio.models.decoder.ctc_decoder</code> to TorchScript so that I can use it in a C++ application. However, I have encountered several issues along the way.</p>
<h4><a name="p-467245-what-i-tried-1" class="anchor" href="#p-467245-what-i-tried-1"></a><strong>What I Tried</strong></h4>
<ol>
<li>
<p><strong>Using <code>torch.jit.script()</code></strong></p>
<ul>
<li>Running <code>torch.jit.script(decoder)</code> gives the following error:<pre data-code-wrap="plaintext"><code class="lang-plaintext">torch.jit.frontend.UnsupportedNodeError: Lambda aren't supported
</code></pre>
</li>
<li>The issue seems to be related to a lambda function inside the <code>_get_tokens</code> method of <code>CTCDecoder</code>, which TorchScript does not support.</li>
</ul>
</li>
<li>
<p><strong>Using <code>torch.jit.trace()</code></strong></p>
<ul>
<li>I tried tracing the decoder using:<pre data-code-wrap="python"><code class="lang-python">scripted_decoder = torch.jit.trace(decoder, dummy_input)
</code></pre>
</li>
<li>This resulted in the following error:<pre data-code-wrap="plaintext"><code class="lang-plaintext">RuntimeError: Tracer cannot infer type of [[CTCHypothesis(tokens=tensor([...]))]]
</code></pre>
</li>
<li>The problem here is that the decoder returns a list of <code>CTCHypothesis</code> objects, which TorchScript cannot process.</li>
</ul>
</li>
<li>
<p><strong>Wrapping the Decoder in <code>nn.Module</code></strong></p>
<ul>
<li>I attempted to wrap the decoder inside an <code>nn.Module</code> subclass to make it compatible with scripting, but the same <code>CTCHypothesis</code> type inference error occurred.</li>
</ul>
</li>
</ol>
<h4><a name="p-467245-question-2" class="anchor" href="#p-467245-question-2"></a><strong>Question</strong></h4>
<ul>
<li>Is there any recommended way to export <code>torchaudio.models.decoder.ctc_decoder</code> to TorchScript?</li>
<li>If not, what would be the best approach to deploy it in a C++ application?</li>
<li>Would modifying the source code of <code>CTCDecoder</code> (e.g., replacing lambda functions and using standard PyTorch types) be a feasible solution?</li>
</ul>
<p>Any guidance or workarounds would be greatly appreciated!</p>
<p>Thanks in advance!</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="mention" href="/u/smth">@smth</a> <a class="mention" href="/u/alband">@albanD</a> <a class="mention" href="/u/carolineechen">@carolineechen</a></p>

username: mariaalfaroc
cooked: <p>Hi <a class="mention" href="/u/harald_scheidl">@harald_scheidl</a> !</p>
<p>Do you have any idea?</p>
3D RCNN Implementation

username: Soumya_Kundu
cooked: <aside class="quote" data-post="1" data-topic="157189">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/m2m/48/33611_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/simple-3d-mask-rcnn-implementation/157189">Simple 3D Mask-RCNN Implementation</a> 
  </div>
  <blockquote>
    Hello everyone, 
I am wondering if there is a simple 3D Mask-RCNN code? Also, is there trained models that I can simply just import like ResNet. Please note that I need 3D not 2D. 
Any other state-of-the-art 3D semantic segmentation/Instance segmentation models? 
Thank you very much.
  </blockquote>
</aside>

<p>Exactly the same question, is there someone who can lead me to something relevant? NNDetection does not support it yet and its predecessor is basically put into archive.</p>
<p>Thanks!</p>

username: Sid_1041
cooked: <p>Hi, which pre-trained 3D ResNet model did you use ?</p>
Understanding Optimal T, H, and W for R3D_18 Pretrained on Kinetics-400

username: Sid_1041
cooked: <p>Hi everyone,</p>
<p>I’m working on a <strong>3D CNN</strong> for defect detection. My dataset is such that a single data is a 3D volume (512×1024×1024), but due to computational constraints, I plan to use a sliding window approach** with 16×16×16 voxel chunks as input to the model. I have a corresponding label for each voxel chunk.</p>
<p>I plan to use <strong><a href="https://pytorch.org/vision/0.20/models/generated/torchvision.models.video.r3d_18.html#torchvision.models.video.r3d_18" rel="noopener nofollow ugc">R3D_18</a></strong> (ResNet-3D 18) with <strong>Kinetics-400 pre-trained weights</strong>, but I’m unsure about the settings for the temporal (T) and spatial (H, W) dimensions.</p>
<p><strong>Questions:</strong></p>
<ol>
<li>How should I handle grayscale images with this RGB pre-trained model? Should I modify the first layer from C = 3 to C = 1? I’m not sure if this would break the pre-trained weights and not lead to effective training</li>
<li>Should the T, H, and W values match how the model was pre-trained, or will it cause issues if I use different dimensions based on my data? For me, T = 16, H = 16, and W = 16, and I need it this way (or 32 × 32 × 32), but I want to clarify if this would break the pre-trained weights and prevent effective training.</li>
</ol>
<p>Any insights would be greatly appreciated! Thanks in advance.</p>
The selectable configs for irregular shape when using triton.autotune

username: NiuMa-1234
cooked: <p>I 'm trying to use triton.autotune to accelerate my triton kernel for moe layer. And I found the supported format of the block size in configs in triton’s autotune must be the power of 2. However, in moe layer the shape of the tensor would be irregular.If the block size cannot be devided exactly by tensor’s shape,  a cuda error of ‘illegal memory access’ would be raised. Is there any way to overcome this rule and keep the acceleration? Thank you guys very much.</p>
Transformer Stuck in Local Minima Occasionally

username: martin12
cooked: <p>Hi, I am working on a project to pre-train a custom transformer model I developed and then fine-tune it for a downstream task. I am pre-training the model on an H100 cluster and this is working great. However, I am having some issues fine-tuning. I have been fine-tuning on two H100s using nn.DataParallel in a Jupyter Notebook. When I first spin up an instance to run this notebook (using PBS) my model fine-tunes great and the results are as I expect. However, several runs later, the model gets stuck in a local minima and my loss is stagnant. Between the model fine-tuning how I expect and getting stuck in a local minima I changed no code, just restarted my kernel. I also tried a new node and the first run there resulted in my training loss stuck again the local minima. I have tried several things:</p>
<ol>
<li>Only using one GPU (still gets stuck in a local minima)</li>
<li>Setting seeds as well as CUDA based deterministics:</li>
<li><code>torch.backends.cudnn.deterministic = True</code></li>
<li><code>torch.backends.cudnn.benchmark = False</code></li>
</ol>
<p>At first I thought my training loop was poorly set up, however, running the same seed twice, with a kernel reset in between, yielded the same exact results. I did this with two sets of seeds and the results from each seed matched its prior run. This leads me to be believe something is happening with CUDA in the H100. I am confident my training loop is set up properly and there is a problem with random weight initialization in the CUDA kernel.</p>
<p>I am not sure what is happening and am looking for some pointers. Should I try using a .py script instead of a Notebook? Is this a CUDA/GPU issue?</p>
<p>Any help would be greatly appreciated. Thanks!</p>
Torch.compile CudaGraph creation & downstream systems

username: deepak.vij
cooked: <p>Hi all, looking at the PyTorch code, I understand that torch compiler codegen process first assesses/analyses PyTorch/python function code to see if CudaGraph will work. Here I am assuming that CudaGraph doesn’t support many things that must be checked upfront (heterogeneous/non-Nvidia hardware platform where CudaGraph may not be supported). If it passes that test, codegen creates a runtime wrapper around the generated triton code for applying CudaGraph using “cudagraphify” function in the code [call(args): function in the generated code].</p>
<p>With that as a quick background of my understanding of all this, the question I have is how does all this play out in the downstream inference engines (such as vLLM, SGLang etc.) as they do their own CudaGraph capture/replay at the time of server boot-up process. Do these systems (re)evaluate the applicability of CudaGraph again at the time of CudaGraph capture. In any case, I am just trying to understand the overall Pytorch codegen/CudaGraph process for my education/understanding and clear my brain fog on this topic. Look forward to response from experts on this forum. Appreciate it. Thanks.</p>
<p>Regards,<br>
Deepak Vij</p>

username: richard
cooked: <p>vLLM handles its cuda-graphing separately from torch.compile right now. So it does not use torch.compile’s built-in cudagraphs.</p>

username: deepak.vij
cooked: <p>Makes sense. Thanks.</p>
Couldn't find the retrieval_recall in torcheval

username: songsong0425
cooked: <p>Hi, I’m not sure it is appropriate for this forum, but I have a problem with the archival.<br>
When I tried to use the <code>retrieval_eval</code> from torcheval, I couldn’t run it because it didn’t exist in the package.<br>
<a href="https://pytorch.org/torcheval/main/generated/torcheval.metrics.functional.retrieval_recall.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/torcheval/main/generated/torcheval.metrics.functional.retrieval_recall.html</a></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c90944d4d2e1934cd171736ed8ce0d1c93ffa37.png" data-download-href="https://discuss.pytorch.org/uploads/default/0c90944d4d2e1934cd171736ed8ce0d1c93ffa37" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/c/0c90944d4d2e1934cd171736ed8ce0d1c93ffa37.png" alt="image" data-base62-sha1="1N9ulab9UolWPeDb7huhWqBj8j5" width="690" height="95" data-dominant-color="F9E0E1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1438×199 8.22 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Does anyone know the proper version of torcheval for it?<br>
My torcheval version is 0.0.7.</p>

username: ptrblck
cooked: <p><code>retrieval_recall</code> was added in Nov. 2023 in <a href="https://github.com/pytorch/torcheval/pull/186">this PR</a>. However, it seems the wheels were not published to PyPI after Aug. 2023 as seen <a href="https://pypi.org/project/torcheval/#files">here</a> so you might need to build from source.</p>

username: songsong0425
cooked: <p>Thank you for your prompt reply!<br>
Then, can I ask one more question? (Also, if it is not proper for this forum, please let me know.)</p>
<p>Instead of using torcheval, I tried to use torchmetrics for the same function, RetrievalRecall.<br>
<a href="https://lightning.ai/docs/torchmetrics/stable/retrieval/recall.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://lightning.ai/docs/torchmetrics/stable/retrieval/recall.html</a><br>
While I tried some example code, it only returned one tensor by aggregating values (e.g., mean, min, max, median).<br>
How can I get the Recall@K value ‘per’ index?<br>
Thank you for reading this question.</p>
Debugging slow TorchDynamo Cache Lookup

username: nhuck
cooked: <p>We have a model that is slowed down occasionally. It spends 847 ms in __get_attr__. I am not sure why or how to begin debugging.</p>
<p>Any assistance would be much appreciated. Thanks!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f3ca5500b2e4149187796733e635c162f124f4b8.png" data-download-href="https://discuss.pytorch.org/uploads/default/f3ca5500b2e4149187796733e635c162f124f4b8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3ca5500b2e4149187796733e635c162f124f4b8_2_689x373.png" alt="image" data-base62-sha1="yMFwdNKmQCtMmXTaHMyKme65luE" width="689" height="373" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3ca5500b2e4149187796733e635c162f124f4b8_2_689x373.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/3/f3ca5500b2e4149187796733e635c162f124f4b8_2_1033x559.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/3/f3ca5500b2e4149187796733e635c162f124f4b8.png 2x" data-dominant-color="BAB3B2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1346×729 126 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: richard
cooked: <p>that doesn’t seem good, do you have a script we could run to reproduce this?</p>
Autograd with mutable custom operator

username: BeachHut
cooked: <p>The <a href="https://pytorch.org/tutorials/advanced/cpp_custom_ops.html" rel="noopener nofollow ugc">Custom C++ and CUDA Operators tutorial</a> says “Do not return any mutated Tensors as outputs of the operator”. How can I add autograd/backward support for the operator when the outputs are not returned (and so I do not think there will be a corresponding <code>grad</code> input to the backward function)?</p>

username: BeachHut
cooked: <p>It seems that you need to call the <code>.clone()</code> method of the mutated inputs and return those. This allows you to return them (and so have corresponding <code>grad</code> inputs to the backward function) while also not violating the rule of not returning mutated Tensors (as you are returning clones rather than the actual mutated Tensors).</p>
Build instructions for ROCM

username: Matthias_Moller
cooked: <p>I wonder if it would be possible to share the build instructions/configurations for the official ROCM releases of LibTorch (the C++ API not PyTorch). Also, and that is the main reason why I need to compile LibTorch from source, would it be possible to compile future releases with MPI/Distributed enabled?</p>
<p>Thanks</p>

username: fluidnumerics_joe
cooked: <p>Hey <a class="mention" href="/u/matthias_moller">@Matthias_Moller</a> ,<br>
I know I’m a bit late to the game here, but I found this on the ROCm fork of Pytorch that should address the first part of your question <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/rocm/pytorch/blob/main/docs/libtorch.rst">
  <header class="source">

      <a href="https://github.com/rocm/pytorch/blob/main/docs/libtorch.rst" target="_blank" rel="noopener nofollow ugc">github.com/rocm/pytorch</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/rocm/pytorch/blob/main/docs/libtorch.rst" target="_blank" rel="noopener nofollow ugc">docs/libtorch.rst</a></h4>

<div class="git-blob-info">
  <a href="https://github.com/rocm/pytorch/blob/main/docs/libtorch.rst" rel="noopener nofollow ugc"><code>main</code></a>
</div>


      <pre><code class="lang-rst">libtorch (C++-only)
===================

The core of pytorch does not depend on Python. A
CMake-based build system compiles the C++ source code into a shared
object, libtorch.so.

AMD ROCm Support
------------------------------
If you're compiling for AMD ROCm then first run this command:
::
   cd &lt;pytorch_root&gt;

   # Only run this if you're compiling for ROCm
   python tools/amd_build/build_amd.py

Additional information about ROCm support can be found in the top-level
`README &lt;https://github.com/pytorch/pytorch/blob/main/README.md&gt;`_.

Building libtorch using Python
</code></pre>



  This file has been truncated. <a href="https://github.com/rocm/pytorch/blob/main/docs/libtorch.rst" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>This covers building libtorch for AMD GPUs in the top section. I’m planning on giving this a try.</p>
<p>For your second question, are you wanting the official ROCm release built with MPI enabled ? That might be a question best opened on an issue at <a href="https://github.com/ROCm/ROCm/issues" class="inline-onebox" rel="noopener nofollow ugc">GitHub · Where software is built</a></p>

username: Matthias_Moller
cooked: <p>Thanks very much <a class="mention" href="/u/fluidnumerics_joe">@fluidnumerics_joe</a>. I’ll make an issue about building the official ROCm release with MPI enabled.</p>