akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Synthetic YOLO Dataset Generator

Igor1: <p>Hello!<br>
I’m excited to share a new Unity-based tool I’ve been working on: Synthetic YOLO Dataset Generator<br>
<a href="https://assetstore.unity.com/packages/tools/ai-ml-integration/synthetic-yolo-dataset-generator-325115" rel="noopener nofollow ugc">https://assetstore.unity.com/packages/tools/ai-ml-integration/synthetic-yolo-dataset-generator-325115</a></p>
<p>It automatically creates high-quality synthetic datasets for object detection and segmentation tasks in the YOLO format. If you’re training computer vision models (e.g. with YOLOv5/YOLOv8) and struggling to get enough labeled images, this might help! <img src="https://discuss.pytorch.org/images/emoji/apple/tada.png?v=14" title=":tada:" class="emoji" alt=":tada:" loading="lazy" width="20" height="20"></p>
<p>What it does: Using the Unity engine, the tool spawns 3D scenes with random objects, backgrounds, lighting, etc., and outputs images with bounding box annotations (YOLO txt files) and segmentation masks. You can generate thousands of diverse training images without manual labeling. It’s like a virtual data factory – great for augmenting real datasets or getting data for rare scenarios.</p>
<p>How it helps: Synthetic data can improve model robustness. For example, I used this generator to create a dataset of 5k images for a custom object detector, and it significantly boosted my model’s accuracy in detecting products on shelves. It’s useful for researchers (to test hypotheses quickly), engineers (to bootstrap models before real data is available), or hobbyists learning YOLO/CV (to experiment with models on custom data).</p>
<p>See it in action: I’ve made a short demo video showing the generator in action – YouTube Demo: <a href="https://youtu.be/lB1KbAwrBJI" rel="noopener nofollow ugc">https://youtu.be/lB1KbAwrBJI</a>.</p>
ValueError: Unsupported ONNX opset version: 21

hitbuyi: <p>my environment is as below</p>
<pre><code class="lang-auto">torch  2.2.2
onnx  1.18.0
onnxruntime  1.20.0
</code></pre>
<p>when I exporot onnx model</p>
<pre><code class="lang-auto">      torch.onnx.export(
            post_model,
            rpn_input,
            args.save_onnx,
            export_params=True,
            opset_version=21,
            do_constant_folding=True,
            keep_initializers_as_inputs=False,
            input_names=['input'],
            output_names=output_names,
        )
</code></pre>
<p>it prompts error</p>
<pre><code class="lang-auto">2025-08-01 14:47:28,899   INFO  ==&gt; Done (loaded 201/201)
[INFO] Exporting 2 detection heads with heads: ['center', 'height', 'dim', 'rot']
[INFO] Exporting ONNX model to models/pcdet_neck_head.onnx, dtype=float32
/home/hitbuyi/anaconda3/envs/pt222/lib/python3.10/site-packages/torch/onnx/utils.py:1548: OnnxExporterWarning: Exporting to ONNX opset version 20 is not supported. by 'torch.onnx.export()'. The highest opset version supported is 17. To use a newer opset version, consider 'torch.onnx.dynamo_export()'. Note that dynamo_export() is in preview. Please report errors with dynamo_export() as Github issues to https://github.com/pytorch/pytorch/issues.
  warnings.warn(
Traceback (most recent call last):
  File "/home/hitbuyi/AD_Projects/Lidar/centerpoint_deploy/centerpoit_export/export_neck_head.py", line 195, in &lt;module&gt;
    main()
  File "/home/hitbuyi/AD_Projects/Lidar/centerpoint_deploy/centerpoit_export/export_neck_head.py", line 173, in main
    torch.onnx.export(
  File "/home/hitbuyi/anaconda3/envs/pt222/lib/python3.10/site-packages/torch/onnx/utils.py", line 516, in export
    _export(
  File "/home/hitbuyi/anaconda3/envs/pt222/lib/python3.10/site-packages/torch/onnx/utils.py", line 1576, in _export
    GLOBALS.export_onnx_opset_version = opset_version
  File "/home/hitbuyi/anaconda3/envs/pt222/lib/python3.10/site-packages/torch/onnx/_globals.py", line 59, in export_onnx_opset_version
    raise ValueError(f"Unsupported ONNX opset version: {value}")
ValueError: Unsupported ONNX opset version: 21
</code></pre>
<p>Does pytorch 2.2.2 + onnx 1.18.0+ onnruntime 1.20.0 support opset 21?</p>

ptrblck: <p>Could you try to execute your code with the most recent PyTorch release to see if this opset is already supported?</p>
Strange random PyTorch behavior

ptrch_c_m: <p>Suddenly, even though I don’t use any seed, the initialization weights of a Linear layer and the order of the training batches are the same between different individual runs of the same script.</p>
<p>Is there a way to check why this is happening? Is there a way to disable all seeds in PyTorch?</p>

valerian.rey: <p>Are you using pure PyTorch or do you also use a library like PyTorch lightning? If so, the library could be fixing the seed somewhere.</p>

ptrch_c_m: <p><a class="mention" href="/u/valerian.rey">@valerian.rey</a> thank you for your reply. I indeed use <code>Lightning</code>, but that happened suddenly. I have run that script several times with the expected behavior. Even though this forum is for pure <code>PyTorch</code>, do you have any idea how can I check if <code>Lightning</code> does that?</p>

valerian.rey: <p>Maybe check if any of the functions from <a href="https://pytorch-lightning.readthedocs.io/en/1.7.7/api/pytorch_lightning.utilities.seed.html#pytorch_lightning.utilities.seed.seed_everything" rel="noopener nofollow ugc">here</a> (in particular <code>seed_everything</code>) is called (it could be in your code or in some code that you import).</p>
<p>It could also be completely unrelated to Lightning. For example a call to <code>torch.manual_seed</code> in some code that you import.</p>

ptrch_c_m: <p>It was an import that apparently did not have to do anything with that. Thank you for your help.</p>
Cannot install torch2.7.0 at centos7

Somezak1: <p>I want to use vllm 0.9.0 since its new feature support running fp8 like models in A100. But vllm 0.9.0 strongly related to pytorch 2.7.0. I cannot install pytorch 2.7.0 at Centos7 since my linux machine’s glibc is 2.17, and pytorch 2.7.0 requires glibc 2.28.</p>
<p>Any guys can help me ?</p>

ptrblck: <p>CentOS7 reached its EOL on June 30 2024 and we thus bumped the min. GLIBC to 2.28. I would recommend updating your OS to receive security updates etc. Alternatively, you could try to install an older PyTorch binary but it might not be compatible with the desired vllm binary.</p>
Weight_norm only after initializing layer weights

mihneasteiu: <p>An LLM just pointed out to me that calling <code>torch.nn.utils.parametrizations.weight_norm</code> should only be done AFTER calling <code>torch.nn.init</code> for weight initialization, or otherwise the <code>torch.nn.init</code> call doesn’t affect the weights of the layer. How does it know this? I wasn’t able to find a source online mentioning this. Thanks!</p>

ptrblck: <p>From the <a href="https://docs.pytorch.org/docs/stable/generated/torch.nn.utils.parametrizations.weight_norm.html">docs</a>:</p>
<blockquote>
<p>Weight normalization is a reparameterization that decouples the magnitude of a weight tensor from its direction. This replaces the parameter specified by <code>name</code> with two parameters: one specifying the magnitude and one specifying the direction.</p>
</blockquote>
<p>Assuming you want to initialize the original parameter you should indeed call it before adding the parametrization as otherwise the original parameter will be moved to the <code>originalX</code> attribute as seen in the code example:</p>
<pre data-code-wrap="python"><code class="lang-python">m = weight_norm(nn.Linear(20, 40), name='weight')
m
m.parametrizations.weight.original0.size()
m.parametrizations.weight.original1.size()
</code></pre>

Jake_Miller: <p>That’s a great observation. When you apply weight_norm, it wraps the original weights, so if you call torch.nn.init after, it might not affect the intended parameters. It’s usually best to initialize first, then apply weight_norm to ensure your custom initialization is preserved.</p>
How to reduce Memory fragmentation when enable expandable_segments

DongyuXu77: <p>Recently, I’ve been reading the source code of <code>CUDACachingAllocator</code>, and I have a few questions.</p>
<p>First, why is the address space of <code>ExpandableSegment</code> limited to only 1 + 1/8 of the maximum size, instead of a larger value? If we used a larger range, the address space could continuously grow forward, and fragmentation could be addressed via CUDA API remapping.</p>
<p>Second, I’m not quite sure what additional methods can be used to further mitigate memory fragmentation after enabling VMM (Virtual Memory Management), since this approach already seems quite smart and efficient.</p>

DongyuXu77: <p>Thank you for your suggestion, <a class="mention" href="/u/amy598henry">@amy598henry</a>.</p>
<p>After your suggestion, I carefully reviewed the five-level allocation of <code>CUDACachingAllocator</code> and found that once it reaches the fifth level of allocation, which is <code>release_cached_blocks</code>, it introduces significant latency and synchronization operations. Therefore, the most direct approach seems to be avoiding calling the release_cached_blocks function.</p>
<p>I am wondering if simply setting the <code>garbage_collection_threshold</code> and adding more levels of allocation could solve the problem (for example, level 4 currently only releases enough GPU memory to satisfy the requested size; if that still fails, can a new level be added to release more?).</p>
<p>I am not sure if my idea is feasible and hope everyone can provide suggestions.</p>
Execution of multiple model instances concurrently

somashekhargg: <p>Executing multiple Pytorch model instances concurrently for inference in a single process. Two threads in the process invokes inference with input data. No input and output data is shared between the threads. I mean, Device, Thread and CUDA stream synchronization is in place. With this configuration observing inconsistent output. Using each stream for the model.</p>
<p>The issue is happening after upgrading from Libtorch 1.9 to 2.0.</p>
<p>Where as, with Libtorch 2.0,  by running only one instance at a time not observing any issues. Surprisingly in single instace mode performance is also very good.</p>
<p>Would like to know more details/insights on this.<br>
Thank you very much.</p>
<p>Below are the environment details,<br>
GPU - RTX A6000<br>
Libtorch - 2.0 with CUDA 11.8<br>
Windows - 10<br>
Codebase using VS2022<br>
Language - C++<br>
Model type - CNN</p>
<p>Regards<br>
Somashekhar</p>
When writing a torch.autograd.Function, is there some way for the .backward() to know with respect to which input are the gradients being requested?

PaulNaduri: <p>Sorry, that’s a mouthful. Let me try to make it clearer. Suppose I have:</p>
<pre><code class="lang-auto">class MyOP(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input0, input1, input2):
        ...
        return output

    @staticmethod
    def backward(ctx, grad_output):
        ...
        return grad_input0, grad_input1, grad_input2
</code></pre>
<p>If then I do,</p>
<pre><code class="lang-auto">out = MyOP.apply(input0, input1, input2)
grad = torch.autograd.grad(out.sum(), input2)
</code></pre>
<p>can I somehow know in backward() that I only need <code>grad_input2</code> so that I can avoid computing <code>grad_input0</code> and <code>grad_input1</code> and just return <code>None</code> in their place?</p>
<p>As a hacky way of doing it, I could maybe do</p>
<pre><code class="lang-auto">to_be_diff_wrt = [False, False, True] 
out = MyOP.apply(input0, input1, input2, to_be_diff_wrt_to)

class MyOP(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input0, input1, input2, to_be_diff_wrt):
        ctx.save_for_backward(to_be_diff_wrt)
        ...
        return output
</code></pre>
<p>but I’m wondering if there’s a better way.</p>

akrentz6: <p>You can use <code>ctx.needs_input_grad</code> inside <code>backward()</code> to check which inputs actually require gradients. It returns a tuple of booleans denoting whether each of the inputs had requires_grad == True when the op was called (it won’t update if this changes after the operation is called).</p>

PaulNaduri: <p>I don’t think this works in my case because all three inputs require grads and therefore that bool tensor will be [True, True, True]. However, sometimes, I only need to use one or two of the grads and discard the rest, and therefore I would like to avoid computing them because they happen to be quite costly.</p>

Jake_Miller: <p>Interesting question! If all inputs require gradients, <code>ctx.needs_input_grad</code> won’t help in filtering. Maybe a cleaner approach could be passing a flag or metadata through <code>ctx</code> during the forward pass, just like you suggested—it’s not hacky at all, actually quite practical in this case.</p>
Not support 5060

tuqizhao: <ol>
<li>
<p>nvidia 5060， lowest driver is 576.52, latest is 577.00, cuda 12.9  (can’t not down to 12.8 cause has no the driver)<br>
PS C:\Users\Administrator&gt; nvidia-smi.exe<br>
Wed Jul 30 12:11:18 2025<br>
±----------------------------------------------------------------------------------------+<br>
| NVIDIA-SMI 576.52                 Driver Version: 576.52         CUDA Version: 12.9     |</p>
</li>
<li>
<p>latest torch version is  2.7.1 cuda 12.8</p>
</li>
</ol>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.<strong>version</strong><br>
‘2.7.1+cu128’</p>
</blockquote>
</blockquote>
</blockquote>
<p>run failed and get the error：</p>
<p>RuntimeError: CUDA error: device kernel image is invalid</p>
<p>CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.</p>
<p>For debugging consider passing CUDA_LAUNCH_BLOCKING=1</p>
<p>Compile with <code>TORCH_USE_CUDA_DSA</code> to enable device-side assertions.</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/nvidia-geforce-rtx-5090-with-cuda-capability-sm-120-is-not-compatible-with-the-current-pytorch-installation/216518/22">This post</a> might help.</p>
JFI:SSI_with_physisical_AI

Anand_Gupta: <p><img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_tear.png?v=14" title=":smiling_face_with_tear:" class="emoji only-emoji" alt=":smiling_face_with_tear:" loading="lazy" width="20" height="20"></p>
<p>Hey, a new user wants to welcome you with introduction to cache in flux dev model</p>
Replace alias operator in the model

funkysloth: <p>Hi, I have a model which is defined with some alias in the source definition. For example, like “torch.concat” instead of “torch.cat”, or “torch.clip” instead of “torch.clamp”. When exporting in ONNX, pytorch can “replace” all the alias in some way and the mapping between ATEN and ONNX works.</p>
<p>On the contrary, when exporting to JIT-Trace/Script it doesn’t. This mean that for example “ATEN::concat” stays that way. Next, if you try to convert a ScriptModel (the output of the JIT trace) to ONNX, and that model has some of the alias in the graph, then conversion fails, claiming for example that operators like “ATEN::concat” are not supported.</p>
<p>When calling “torch.onnx.export” with a “nn.Module” instance, the “de-aliasing” transformation is done automatically and export succeeds, if you pass a <code>ScriptModel</code> it doesn’t.</p>
<p>Since I want to keep the output of <code>torch.jit.trace(...)</code> I want to know if there is a way to “de-alias”/transform the model before calling the ONNX export.</p>
<p>I am fine in applying this replacement either at graph level (after the trace) or at <code>nn.Module</code> level, since I have access to the instance. I do NOT have access to the network definition (i.e. the sources).</p>
<p>ChatGPT has proposed two approaches, either operating on the graph or calling torch.fx. Both approach fail at some point (I guess because the network is more complex than a toy example).</p>
<p>Can you recommend a valid approach?</p>
GPU Memory Management

Flex: <p>Hi, I wrap a model in a customized class called Agent. I use a for loop to iterate over questions. WIthin the for loop, I need to create different instances of Agent class. Model is the same, but some variables within Agent class are different. I notice that as the number of iteration increases, the GPU memory increases. A small code snippet to reproduce this phenomenon is:</p>
<pre><code class="lang-auto">from transformers import AutoModelForCausalLM 
for _ in range(10):
  model = AutoModelForCausalLM("meta-llama/Meta-Llama-3-70B-Instruct", device_map="auto")
  # trial 1
  del model
  torch.cuda.empty_cache()
</code></pre>
<p>I tried <code>del model</code> + <code>torch.cuda.empty_cache()</code>, but GPU usage still keeps increasing after each iteration. Is there an appropriate way to avoid an increase in GPU usage?</p>
FullyShardedDataParallel hangs depending

michael4: <ol>
<li>Why do I have to wrap torch.nn.Embedding when using distributed backend=“nccl”? If I comment out line in custom_wrap_policy specifying torch.nn.Embedding then FSDP will hang. Wrapping torch.nn.Embedding does not hang. (Using “gloo” it will not hang either with or withoug torch.nn.Embedding)</li>
</ol>

William_Gram: <p>FullyShardedDataParallel can hang due to mismatched GPU setup deadlocks or uneven batch sizes. Check your setup batch sizes and use find_unused_parametersTrue if needed.</p>
How to address pretrained facenet overfitting for facial verification?

Jayasuryan: <p>Hello everyone,<br>
I’m currently working on a building a facial verification system using facenet-pytorch. I would like to ask for some guidance with this project as I have observed my model was overfitting. I will be explaining how the dataset was configured and my approach to model training below:</p>
<ol>
<li>Dataset Setup</li>
</ol>
<ul>
<li>Gathered a small dataset containing 328 anchor images and 328 positive images of myself, 328 negative images (taken from lfw dataset).</li>
<li>Applied transforms such as resize(160,160),random horizontal flip, normalization.</li>
</ul>
<ol start="2">
<li>Training configuration</li>
</ol>
<ul>
<li>batch_size = 16</li>
<li>learning_rate = 1e-4</li>
<li>patience for early stopping = 10</li>
<li>epochs = 50</li>
<li>mixed precision training (fp16)</li>
<li>loss = TripletMarginLoss(margin=0.5)</li>
<li>optimizer = AdamW</li>
<li>learning rate scheduler = exponential scheduler</li>
</ul>
<ol start="3">
<li>Training approach</li>
</ol>
<ul>
<li>Initially all the layers in the facenet were frozen except last_linear layer.</li>
<li>I proceeded to train the network.</li>
<li>I observed the model was overfitting as the training loss was able decrease monotonically, while the validation loss fluctuated.</li>
</ul>
<ol start="4">
<li>Solutions I tried</li>
</ol>
<ul>
<li>I have tried the same approach using a larger dataset where I had over 6000 images.</li>
<li>The results were the same, the model was still overfitting. I did not observe any difference that adding more data would help address overfitting.</li>
</ul>
<p>I will be attaching the code below for reference:<br>
<a href="https://colab.research.google.com/drive/1SDY8MKAXZ2aKl10C8yAGIBqUUnYsf0LD?usp=sharing" rel="noopener nofollow ugc">colab notebook</a></p>
<p>I would appreciate any suggestions that can be provided on how I can address:</p>
<ul>
<li>Improving generalization with respect to validation error.</li>
<li>What are the best practices to follow when finetuning facenet with triplet loss ?</li>
<li>Is there any sampling strategies that I need to try while sampling the triplet pairs for training ?</li>
</ul>
<p>Thanks in advance for your help !</p>
Ddp training and eval question

Julius_Lee: <p>Hi all,<br>
I’m pretty new to using pytorch. So, please let me know if there’s anything unclear.</p>
<p>I’ve been working on a project to train a BERT4Rec model using DDP.</p>
<p>Since our input data are many rows spanning across many files, I decided to implement a rank and world_size aware iterative dataset that pretty much just distribute yield rows round robin based on rank and world_size, and not using DistributedDataSampler.</p>
<p>My model is wrapped with DDP and works perfectly with training.<br>
For eval, it works well with smaller batch numbers.<br>
However, with larger batch numbers (like 1000 batches * 1024 batch_size)</p>
<p>After it goes through eval loop, it falls out of sync before I could do call all_gather to consoladate the results. I had a barrier after loop ends and what I saw was that a couple ranks passed the barrier first without waiting while the rest kept waiting for them on the barrier. Thie ends of hanging for 30mins before timing out.</p>
<p>There is no synchronization operation inside eval loop, just forward pass and calculating eval metrics for each batch in each rank.</p>
<p>I’ve tried many ways to eliminate possible mistakes:</p>
<ul>
<li>Eliminate data imbalance - i tried drop_last = True and manual max index loop break condition</li>
<li>torch.barrier() after the eval loop ends to make sure all ranks finish the loops before all_gather calls</li>
<li>Made sure ranks are still in sync after training loop</li>
</ul>
<p>In the end, the only thing that worked was to unwrap the model to use it for the eval loop with<br>
<code>model = model.module</code></p>
<p>I searched everywhere to see if this is something that is correct, but I’ve not found much about it.<br>
Would like to hear if anyone has encountered this issue or have some idea whether or not to do it this way.</p>
<p>Thanks!!</p>

H-Huang: <p>Hi, if you can get the call stack of there the model is hanging that would give more actionable feedback, you can inspect the process with <a href="https://github.com/benfred/py-spy" rel="noopener nofollow ugc">py-spy</a></p>
<p>One hypothesis I have is that during the forward pass of eval, DDP is doing a broadcast for the buffers in your module (<a href="https://docs.pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#:~:text=single%2Ddevice%20modules" class="inline-onebox" rel="noopener nofollow ugc">DistributedDataParallel — PyTorch 2.7 documentation</a>)-,broadcast_buffers,-(bool)%20%E2%80%93%20Flag). Are you running eval using <code>with torch.no_grad():</code>. You can also try setting <code>broadcast_buffers=False</code></p>

Julius_Lee: <p>Thanks for the reply.<br>
I’m running on sagemaker training job so won’t be able to run pyspy. I think it has its own profile dashboard but not sure if it would show up since it would just hang.<br>
Eval loop is using torch.no_grad.<br>
I will try running it with broadcast_buffers=False and report back. Thanks for pointing it out, I didn’t know forward pass still requires some ddp communication.</p>
Continued pre-training large models with FSDP2?

hpcpony: <p>Is there a pattern for doing continued pre-training of large models with FSDP2?  I don’t see a feature like the “sync_module_states” from FSDP1.  Is it implicit in FSDP2?</p>

hpcpony: <p>I think I found my answer.  It looks like with FSDP2 this is handled by using the Distributed Checkpoint capabilities (torch.distributed.checkpoint).</p>
<p>There some tutorial at <a href="https://docs.pytorch.org/tutorials/intermediate/FSDP_tutorial.html#state-dict-with-dcp-apis" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Fully Sharded Data Parallel (FSDP2) — PyTorch Tutorials 2.7.0+cu126 documentation</a></p>
<p>(correct me if I’m wrong)</p>

H-Huang: <p>Yep FSDP2 uses dtensors which DCP can also handle. Also recommend looking at the checkpoint example in torchtitan (<a href="https://github.com/pytorch/torchtitan/blob/f3e2a75e151872b3dc346d3bfa37d121e413242d/torchtitan/train.py#L298" class="inline-onebox" rel="noopener nofollow ugc">torchtitan/torchtitan/train.py at f3e2a75e151872b3dc346d3bfa37d121e413242d · pytorch/torchtitan · GitHub</a>)</p>
Why would functional and non-functional broadcast use `src` with different semantics?

hankzhwang: <p>Hi, I am wondering why the <code>src</code> means differently in <code>torch.distributed._functional_collectives.broadcast</code> and <code>torch.distributed.distributed_c10d.broadcast</code>. In functional version, <code>src</code> is the local rank inside the group, while in the non-functional version, <code>src</code> is a global rank. This is kind of misleading to users that first use functional version.</p>

H-Huang: <p>Agree that is not ideal, but because <code>torch.distributed.distributed_c10d.broadcast</code> is the original collective we can’t change it for backwards compatibility reasons. The functional version must always pass in a group so it can use the local rank</p>
How to Speed Up PyTorch Model in CPU Eager Mode?

Vaen_Joe: <p>Hi, I’m currently running a PyTorch model in <strong>eager mode</strong> on the <strong>CPU</strong>, and I’m looking for ways to <strong>improve inference performance</strong>.</p>
<p>The model is already using typical linear layers and tensor operations, but performance seems suboptimal compared to expectations. I’m not using TorchScript or <code>torch.compile</code> in this context — this is purely eager execution.</p>
<p>I would like to know:</p>
<ul>
<li>How can I <strong>ensure PyTorch is utilizing all CPU threads</strong> efficiently?</li>
<li>Are there any <strong>recommended environment variables</strong> or settings (e.g., MKL, OpenMP) to tweak?</li>
<li>What are the <strong>best practices</strong> to maximize CPU performance in eager mode?</li>
</ul>
<p>Here’s what I’ve tried so far:</p>
<p>python</p>
<pre><code class="lang-auto">import torch
torch.set_num_threads(4)
torch.set_num_interop_threads(4)
</code></pre>
<p>And I also set:</p>
<p>bash</p>
<pre><code class="lang-auto">export OMP_NUM_THREADS=4
export MKL_NUM_THREADS=4
</code></pre>
<p>But the speedup was marginal. Are there any other profiling tools or CPU-specific optimizations you would recommend?</p>
<p>Thanks!</p>
optim.Adam error

mohmead: <p>i get</p>
<p>File :1<br>
def <strong>create_fn</strong>(<strong>dataclass_type_nx-loopback</strong>, <strong>dataclass_HAS_DEFAULT_FACTORY</strong>, <strong>dataclass_builtins_object</strong>, <strong>dataclass_return_type</strong>):<br>
^<br>
SyntaxError: invalid syntax<br>
when use any optimizer  such that<br>
optimizer = optim.Adam(params = params, lr = 0.001)</p>
<p>Name: dataclasses<br>
Version: 0.6<br>
Summary: A backport of the dataclasses module for Python 3.6<br>
Home-page: <a href="https://github.com/ericvsmith/dataclasses" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ericvsmith/dataclasses</a><br>
Author: Eric V. Smith<br>
Author-email: <a href="mailto:eric@python.org">eric@python.org</a><br>
License: Apache<br>
Location: D:\anaconda3\envs\pytorch\Lib\site-packages<br>
Requires:</p>

ptrblck: <p>Which <code>dataclasses</code> version are you using? Does updating it (in case you are using an older one) help?</p>

mohmead: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="221784">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>dataclasses</code> version</p>
</blockquote>
</aside>
<p>Name: dataclasses<br>
Version: 0.6<br>
Summary: A backport of the dataclasses module for Python 3.6<br>
Home-page: <a href="https://github.com/ericvsmith/dataclasses" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ericvsmith/dataclasses</a><br>
Author: Eric V. Smith<br>
Author-email: <a href="mailto:eric@python.org">eric@python.org</a><br>
License: Apache<br>
Location: D:\anaconda3\envs\pytorch\Lib\site-packages<br>
Requires:</p>

ptrblck: <p>I’m using:</p>
<pre data-code-wrap="python"><code class="lang-python">dataclasses                   0.6
dataclasses-json              0.6.7
</code></pre>
<p>and cannot reproduce any issues:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn 

params = [nn.Parameter(torch.randn(1))]
optimizer = torch.optim.Adam(params = params, lr = 0.001)
</code></pre>
<p>Does this code already fail in your environment?</p>
Should we apply un-normalization before computing the loss?

Eyzzle: <p>Hello everyone, hope you are doing great.</p>
<p>I have a question I imagine to be fairly simple, yet I could not find a direct answer. I am doing forecasting of a single value y, using multiple features X.<br>
Before passing data to my model, I normalize all my features, including the target value <code>y</code> (that we shall call <code>y,scaled</code>).<br>
My model then yields a prediction <code>y_hat,scaled</code>. It is totally possible for me to un-normalize my prediction into <code>y_hat</code>.</p>
<p>The question is the following: should the loss function used by the optimizer be computed  on the scaled or the unscaled data ?<br>
For stability, I was assuming I should compute the loss on scaled data : <code>loss(y,scaled, y_hat,scaled)</code>. Is this the case ? Does it make any difference if I compute instead <code>loss(y, y_hat)</code> ?</p>
<p>Thanks in advance!</p>

ptrblck: <p>One of the main differences in computing the loss using the (un)scaled data could be the gradient magnitude, which could be handled by reducing/increasing the learning rate.<br>
Depending on your use case you might want to compute the “unscaled loss” by unscaling the model output and target to compute the error in your target units in case this interpretation is valid.<br>
E.g. if your model is computing pixel coordinates of keypoints in an image frame it might make sense to train the model to predict normalized outputs in <code>[-1, 1]</code> or <code>[0, 1]</code>. However, to actually print the predicted points and to compute the error in “pixel coordinates” you would unscale them again.</p>
Verifying grad_x after autograd double backprop

jvwilliams23: <p>In the process of verifying another piece of code, I need to find <code>x.grad</code> after backpropagating a loss function which involves a gradient penalty (obtained by autograd). Here is an example:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn

x = torch.ones((1, 1, 4, 4)).requires_grad_(True)
conv = nn.Conv2d(in_channels=x.shape[1], out_channels=1, kernel_size=3, padding=1)

y = conv(x)

grads = torch.autograd.grad(
    outputs=y.sum(), inputs=x, create_graph=True, only_inputs=True
)[0]
loss = grads.sum()
loss.backward()
print(conv.weight.grad) # returns something, as expected
print(x.grad)  # is None, but should be **something**?
</code></pre>
<p>Pretty much, my issue/question is shouldn’t <code>x.grad</code> have some value? It is used as the input to <code>grads</code>, which is then used as a loss.</p>
<p>How do I actually obtain the input gradient for this case (for downstream verification purposes)?</p>

valerian.rey: <p>Hi! <code>grads</code> is the gradient of <code>y.sum()</code> with respect to <code>x</code>, which actually only depends on <code>conv.weight</code>. You can see that by fixing the values of <code>conv.weight</code> (for example by using <code>torch.manual_seed(0)</code> before instantiating it) and varying the values of <code>x</code>: you’ll observe always the same value for <code>grads</code>.</p>
<p>So, since <code>grads</code> is independent of <code>x</code>, it’s normal to have <code>x.grad</code> set to <code>None</code> after the call to <code>loss.backward()</code>.</p>
<p>Note that if you compute <code>grads</code> with respect to the parameters rather than <code>x</code>, it will become dependent on the value of <code>x</code> (and independent of the value of the parameters), and so <code>x.grad</code> will be populated but not <code>conv.weight.grad</code>. Maybe this is what you intended to do?</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn

x = torch.ones((1, 1, 4, 4)).requires_grad_(True)
conv = nn.Conv2d(in_channels=x.shape[1], out_channels=1, kernel_size=3, padding=1)

y = conv(x)

grads = torch.autograd.grad(outputs=y.sum(), inputs=list(conv.parameters()), create_graph=True, only_inputs=True)[0]  # Change in the inputs here
loss = grads.sum()
loss.backward()
print(conv.weight.grad) # None
print(x.grad)  # something
</code></pre>
<p>Lastly, the <code>only_inputs</code> parameter of <code>torch.autograd.grad</code> is deprecated so you can get rid of it (source: 2nd note of  <a href="https://docs.pytorch.org/docs/stable/generated/torch.autograd.grad.html#torch-autograd-grad" class="inline-onebox" rel="noopener nofollow ugc">torch.autograd.grad — PyTorch 2.7 documentation</a>).</p>
<p>I hope this helps! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Torch::cuda::is_available() always returns false

marcleon: <p>Good morning,</p>
<p>I’m developing a c++ application using libtorch2.7.1+cu128. I cannot send the model to the GPU because the method is_available() form torch::cuda always returns false.</p>
<p>The dll’s I’m using are the following ones:</p>
<p>asmjit.dll<br>
cublasLt64_12.dll<br>
cublas64_12.dll<br>
cudart64_12.dll<br>
cudnn64_9.dll<br>
cufft64_11.dll<br>
cupti64_2025.1.0.dll<br>
curand64_10.dll<br>
cusolver64_11.dll<br>
cusparse64_12.dll<br>
c10.dll<br>
c10_cuda.dll<br>
fbgemm.dll<br>
libiomp5md.dll<br>
nvrtc64_120_0.dll<br>
torch.dll<br>
torch_cpu.dll<br>
torch_cuda.dll</p>
<p>and the lib files are the following ones:</p>
<p>asmjit.lib"<br>
fbgemm.lib"<br>
c10.lib"<br>
c10_cuda.lib"<br>
torch.lib"<br>
torch_cpu.lib"<br>
torch_cuda.lib"</p>
<p>I’m taking this files from the debug and release official folder I’ve downloaded from official website. I’ve done a cmake file named findTorch.cmake where I load the .lib files and copy de dll’s to the binary distribution folder.</p>
<p>Can someone help me to solve this?</p>
<p>Thanks! Kind regards!</p>

qyq: <p>Hello, I encounter the same question when I use <strong>libtorch1.9.1</strong> in <strong>visual studio 2019</strong>(windows).<br>
I ensure:<br>
(1) I have added all the relevant include and lib in the project<br>
(2) The version of the libtroch is 1.9.1<br>
(3) The CUDA environment is ok and matched, which is cu111<br>
(4) The pytorch cuda is avaibale but the libtorch is not<br>
(5) The environment varibles of CUDA or Libtorch have added to the system path<br>
When I check the output of the visual studio 2019, I found the <strong>torch_cpu.dll</strong> and <strong>c10.dll</strong> is loaded, but no torch_cuda.dll. So, I guess that is the reason.<br>
But I can not slove the problem</p>
How to use flight recorder in a convenient way?

zzibc: <p>Hi, all!</p>
<p>Is there a more convenient way to use the Flight Recorder?</p>
<p>I followed the example in the documentation at <a href="https://docs.pytorch.org/tutorials/prototype/flight_recorder_tutorial.html" rel="noopener nofollow ugc">https://docs.pytorch.org/tutorials/prototype/flight_recorder_tutorial.html</a> and ran into an error when executing <code>torchfrtrace</code>:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/opt/conda/bin/torchfrtrace", line 5, in &lt;module&gt;
    from tools.flight_recorder.fr_trace import main
ModuleNotFoundError: No module named 'tools.flight_recorder'
</code></pre>
<p>As a workaround, I had to manually copy the <code>tools/flight_recorder</code> directory from the PyTorch source code and add it to my <code>PYTHONPATH</code>. Is there a more straightforward or convenient way to use the Flight Recorder without having to do this?</p>
Object detection for extremely small object

AnhMinhTran: <p>Hi all,</p>
<p>I’m looking for any pre processing technique that can help with detecting very small object in pictures, for example.</p>
<p>If I just put the annotation in, the results are very poor</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee844911abe7913d3f7518de69b77a5e903055f8.png" data-download-href="https://discuss.pytorch.org/uploads/default/ee844911abe7913d3f7518de69b77a5e903055f8" title="12"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee844911abe7913d3f7518de69b77a5e903055f8_2_690x466.png" alt="12" data-base62-sha1="y213BF3d6SwFIjqOaXxY7VQJU2A" width="690" height="466" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee844911abe7913d3f7518de69b77a5e903055f8_2_690x466.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee844911abe7913d3f7518de69b77a5e903055f8_2_1035x699.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee844911abe7913d3f7518de69b77a5e903055f8.png 2x" data-dominant-color="5F5E60"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">12</span><span class="informations">1120×757 1.53 MB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

mycul: <p>Hey there!</p>
<p>It’s been a long time since this was originally posted; however, i thought I would add some resources I have found and maybe this kickstarts this topic again.</p>
<ol>
<li>Depending on what model you’re using for these detections, you can get the best anchor sizes for your RPN by refining them to the bounding boxes in your dataset. I have added more in the link here: <a href="https://blog.roboflow.com/what-is-an-anchor-box/" class="inline-onebox" rel="noopener nofollow ugc">What are Anchor Boxes in Object Detection?</a></li>
<li>SAHI (Slicing Aided Hyper Inference): Essentially cuts the image into tons of different, smaller windows and then inferences over them. Paper: <a href="https://arxiv.org/pdf/2202.06934" rel="noopener nofollow ugc">https://arxiv.org/pdf/2202.06934</a></li>
</ol>
<p>That’s all i really have for the moment! If you happen to still be working on this or if anyone reading this has a similar issue, don’t be afraid to reach out!</p>
Help with training performances

Luigi_Scilimati: <p>Hello, newbie here.<br>
I am fighting against my new setup:AMD 9950X, MB is Rog Crosshair X870E, 192GB ram, 2 x samsung 990pro in software raid 1 under linux ubuntu server 24.04LTS. Tried with RTX4090 at first, bought a RTX5090 now and using it.<br>
Training with Pytorch is 1 to 6 it/s with 5090. I reached 9it/s in the past days with RTX4090 but I don’t know how.<br>
Can you please help finding the bottleneck?</p>
<p>python -c “import torch; print(torch.<strong>version</strong>); print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0))”</p>
<p>2.8.0.dev20250322+cu128<br>
True<br>
NVIDIA GeForce RTX 5090</p>
<p>python --version<br>
Python 3.12.3</p>
<p>Thanks for any feedback you may have.<br>
All the best</p>

ptrblck: <p>I would recommend profiling your code to narrow down where the performance bottleneck is.</p>

CodingWookie: <p>I am having similar experience.</p>
<p>No boost with 5090 and latest nightly torch with 575-open drivers… very odd</p>
Loss.backward() assertion failures on mps

roach: <p>Hello, i seem to be having two different problems trigger on somewhat “normal” shape inputs.<br>
torch version - 2.9.0.dev20250721<br>
MacOS stuff - M3 max , 128gb, version: sequoia 15.5, we’re using MPS.</p>
<p>What i’m trying to do is just a basic 3D Unet. The image shapes we’re passing are 64x256x256.<br>
Issue arises when batch size goes above 8. Using L1 loss (sum reduction) adam opt default values.</p>
<p>What works:<br>
(filters: 3 down :  16 → 128 (at bottleneck)) &gt; Trainable params: 1,401,265, batch size 8.<br>
(filters: 4 down :  16 → 256 (at bottleneck)) &gt; Trainable params: 5,646,385, batch size 8.</p>
<p>What fails:<br>
(filters: 3 down :  16 → 128 (at bottleneck)) &gt; Trainable params: 1,401,265, batch size 16:</p>
<pre><code class="lang-auto"> /AppleInternal/Library/BuildRoots/1c8f7852-1ca9-11f0-b28b-226177e5bb69/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSCore/Types/MPSNDArray.mm:829: failed assertion `[MPSNDArray initWithDevice:descriptor:isTextureBacked:] Error: NDArray dimension length &gt; INT_MAX'
zsh: abort      python local_train_3d_IEC.py
/opt/anaconda3/envs/dlops_v10/lib/python3.12/multiprocessing/resource_tracker.py:255: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
</code></pre>
<p>(filters: 3 down :  16 → 128 (at bottleneck)) &gt; Trainable params: 1,401,265, batch size 32:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/Users/petros/Documents/git/roo_archives/deep_learning/projects/denoise_baseline_IEC/local_train_3d_IEC.py", line 163, in &lt;module&gt;
    loss.backward()
  File "/opt/anaconda3/envs/dlops_v10/lib/python3.12/site-packages/torch/_tensor.py", line 625, in backward
    torch.autograd.backward(
  File "/opt/anaconda3/envs/dlops_v10/lib/python3.12/site-packages/torch/autograd/__init__.py", line 354, in backward
    _engine_run_backward(
  File "/opt/anaconda3/envs/dlops_v10/lib/python3.12/site-packages/torch/autograd/graph.py", line 829, in _engine_run_backward
    return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: srcNDArray INTERNAL ASSERT FAILED at "/Users/runner/work/pytorch/pytorch/pytorch/aten/src/ATen/native/mps/OperationUtils.mm":517, please report a bug to PyTorch. 
</code></pre>
<p>These happen for (filters: 4 down :  16 → 256 (at bottleneck)) &gt; Trainable params: 5,646,385, and the respective batch sizes (16,32).<br>
I assume that something goes above the signed int max (2147483647), however, i’m not sure if it’s the total number of things in the input data tensor, or the backprop graph having a meltdown.</p>
<p>Can someone explain what’s happening?</p>
Why does init_device_mesh() or DeviceMesh() have to be called globally?

githubsgi: <p>Found this odd behavior of DeviceMesh()/init_device_mesh().   When they are called  from inside a function, the following errors are observed.</p>
<p>Duplicate GPU detected : rank 1 and rank 0 both on CUDA device</p>

yf225: <p>Do you have an error stack?</p>
<p><a class="mention" href="/u/tianyu">@tianyu</a> <a class="mention" href="/u/irisz">@irisz</a> wonder is this a known limitation?</p>

fduwjj: <p>The Process Group creation API inside require all ranks to call that’s why we require that.</p>

irisz: <p>This should be not related to whether they are called inside a function, but whether <code> torch.cuda.set_device(rank)</code> has been set before calling <code>DeviceMesh()/init_device_mesh()</code>. Similar issue: <a href="https://github.com/pytorch/pytorch/issues/129385" class="inline-onebox" rel="noopener nofollow ugc">Importing torch.utils.benchmark causes torch.distributed to "Duplicate GPU detected" · Issue #129385 · pytorch/pytorch · GitHub</a><br>
<a href="https://github.com/pytorch/torchrec/issues/328" class="inline-onebox" rel="noopener nofollow ugc">NCCL error while instantiating DistributedModelParallel · Issue #328 · pytorch/torchrec · GitHub</a></p>
<p>Could you give this a try and see if works?</p>
<p>Update: As for why <code>init_device_mesh</code> or <code>DeviceMesh()</code> have to be called globally, as <a class="mention" href="/u/fduwjj">@fduwjj</a> answered above, <code>init_device_mesh</code> or <code>DeviceMesh()</code> would create ProcessGroup creation underneath and we do require all ranks to participate.</p>
Conda - Huge disk usage

benoriol: <p>I just saw that my conda environment is 13GB in size. I looked up which packages are taking more space and they are mostly pytorch / cuda modules.</p>
<p>I wonder if that’s normal or if there’s any way to avoid using so much disk space.</p>
<p>This is the list of the modules that take more space in my environment and their respective sizes.</p>
<pre><code class="lang-auto">
pytorch-1.13.0-py3.10_cuda11.7_cudnn8.5.0_0.json:  "size":  1229942678,
nsight-compute-2022.3.0.22-0.json:                 "size":  639598244,
libcublas-dev-11.11.3.6-0.json:                    "size":  413253930,
libcublas-11.11.3.6-0.json:                        "size":  381637889,
libcusparse-dev-11.7.5.86-0.json:                  "size":  377165256,
libcufft-dev-10.9.0.58-0.json:                     "size":  289240757,
mkl-2021.4.0-h06a4308_640.json:                    "size":  229783051,
libcusparse-11.7.5.86-0.json:                      "size":  184891885,
libnpp-11.8.0.86-0.json:                           "size":  154995740,
libnpp-dev-11.8.0.86-0.json:                       "size":  151472361,
libcufft-10.9.0.58-0.json:                         "size":  149741913,
cuda-nvvp-11.8.87-0.json:                          "size":  119905249,
cuda-nsight-11.8.86-0.json:                        "size":  119143043,
libcusolver-11.4.1.48-0.json:                      "size":  101143771,
</code></pre>
<p>I used <code>grep '"size":' ${CONDA_PREFIX}/conda-meta/*.json | sort -k3rn | sed 's/.*conda-meta\///g' | column -t</code>to output the previous list.</p>
<p>Thanks,<br>
B.</p>

ptrblck: <p>I’m unsure how the <code>13GB</code> are calculated as your output shows ~3-4GB of usage.<br>
In any case, the issue is also tracked <a href="https://github.com/pytorch/builder/issues/1185">here</a> and <a href="https://github.com/pytorch/builder/commit/942c85481eb9f3c67aafd21bb3292202d4f1c4ef">this PR</a> should already reduce the size.</p>

benoriol: <p>The 13GB were calculated with: <code>du -sh miniconda3/envs/env_name </code>.</p>
<p>The output I pasted is not the whole output of the command in the initial message, it was just the first few lines (out of 150). I saw that among the ones that took more space there are pytorch and cuda-related stuff. I was suspecting these are taking a large contribution in the 13GB, and wondering if this is normal.</p>
<p>I agree there are other packages that take a lot of space, but it feels like it is always the environments where I use PyTorch with CUDA, that take way more disk space than the rest, whether directly through PyTorch/CUDA or through its dependencies.</p>
<p>Thanks,<br>
B.</p>

Camaro: <p>Have you tried “conda clean -a”?</p>