akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Add more feature in last layer

Joey1: <p>Can i ask, now i’m already add mask predict from unet as channel 4 to image RGB, but my macro f1 still around 69%. Now I want to add more feature like age,location,… in fully connected.</p>

Joey1: <p>thank you so much for your support</p>

J_Johnson: <p>Welcome to the forums. There are a lot of unet models on GitHub. Can you link or copy your current model code?</p>

Joey1: <p>thanks, i can add more feature but my macro still around 85%. Can I ask that is there any other way to increase the macro of the model, now I’m predict benign or malignant using HAM10000.</p>

J_Johnson: <p>I’m not clear on what you’re asking or what model you’re using. Can you provide the code for your model or a sample snippet of code that reproduces the issue?</p>
A question for batch-training RNN

citystrawman: <p>I just started learning NLP and RNN. I read O’Reilly’s <a href="https://www.oreilly.com/library/view/zerokarazuo-rudeep-learning/9784873118369/" rel="noopener nofollow ugc">Deep Learning: natrual language processing</a>, and here’s the Figure showing batch-training in RNN:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/4/440da3b6c9137829f920dbb36869068895b48257.png" data-download-href="https://discuss.pytorch.org/uploads/default/440da3b6c9137829f920dbb36869068895b48257" title="RNN"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/440da3b6c9137829f920dbb36869068895b48257_2_488x500.png" alt="RNN" data-base62-sha1="9I1GOhNccmQUgApCwboRXP9kVHV" width="488" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/440da3b6c9137829f920dbb36869068895b48257_2_488x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/440da3b6c9137829f920dbb36869068895b48257_2_732x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/440da3b6c9137829f920dbb36869068895b48257_2_976x1000.png 2x" data-dominant-color="F5F7F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">RNN</span><span class="informations">978×1002 61.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>（please ignore its Chinese Characters which are not important）</p>
<p>In the above figure, the author uses Truncated BPTT as an example to show how to traing from a 1000-word time series, truncated at every 10 words, with batch size=2.</p>
<p>As the figure shows, the first sequence in first batch from X0, and the second sequence in first batch start from X500, which shift from X0 by 500. So does the other batch.</p>
<p>Then here comes my question: does this mean that all sequences in the first batch has no hidden state to inherit? For example, X500 does not have h499. What is more, does that mean that forward propagating is truncated at X500 (which indicates that the longest memory in this situation could only support 500 words)?</p>

J_Johnson: <p>No. If you’re truncating, the first in the new truncated sequence gets zeros for the hidden state at t0, same as the initial sequence. Then that new hidden state moves on to t1, and so on.</p>

citystrawman: <p>So, can I assume that if we use batch-training with batch number N, we just “separate” the whole texts into N parts which are not “connected“?</p>

J_Johnson: <p>If you’re training set is one continuous document, then yes. But it’s probably a good idea to pad the beginning with a random amount of tokens between each epoch, in order to make those ‘cutoffs’ shifted and variable.</p>
Model.state_dict() doesn't reflect parameter changes after optimizer.step(), but model.parameters() does

andrey1981: <p>Hello!</p>
<p>I have a training loop where parameters are successfully updated when checked via <strong><code>model.parameters()</code></strong>, but <strong><code>model.state_dict()</code></strong> fails to reflect these changes.</p>
<h3><a name="p-474383-working-case-parameters-update-correctly-1" class="anchor" href="#p-474383-working-case-parameters-update-correctly-1"></a><strong>Working Case (Parameters Update Correctly)</strong></h3>
<pre><code class="lang-auto">  class PyTorchTrainer(object):
         def __init__(self,
                      model: PyTorchModel,
                      optim: Optimizer,
                      criterion: _Loss):
             self.model = model
             self.optim = optim
             self.loss = criterion


            def fit(self, X_train: Tensor = None,
                          y_train: Tensor = None)


                    self.optim.zero_grad()
                    output = self.model(X_train.float())
                    loss = self.loss(output, y_train)


                    params_before = list(self.model.parameters())
                    loss.backward()
                    self.optim.step()
                    params_after = list(self.model.parameters())

                    print("Parameters updated:", any(not torch.equal(p1, p2) for p1, p2 in zip(params_before, params_after)))  # True
</code></pre>
<p>where</p>
<pre><code class="lang-auto"> optim: torch.optim.Optimizer      
        X_batch: Tensor       
        loss: torch.nn.CrossEntropyLoss()       
         model extends torch.nn.Module
</code></pre>
<h3><a name="p-474383-problem-case-state_dict-doesnt-reflect-changes-2" class="anchor" href="#p-474383-problem-case-state_dict-doesnt-reflect-changes-2"></a><strong>Problem Case (<code>state_dict()</code> Doesn’t Reflect Changes)</strong></h3>
<pre><code class="lang-auto">   self.optim.zero_grad()
   output = self.model(X_batch.float())
   loss = self.loss(output, y_batch)


   state_before = self.model.state_dict()
   loss.backward()
   self.optim.step()
   state_after = self.model.state_dict()

    print("State dict equal:", state_before == state_after)  # True (unexpected!)
</code></pre>
<p><strong><code>state_dict()</code></strong> should reflect parameter updates after <strong><code>optimizer.step()</code></strong>, just like <strong><code>model.parameters()</code>,</strong>  but it does not</p>
<p>What am I doing wrong?</p>

valerian.rey: <p>It’s not that <code>state_after</code> contains the state before the update, but rather that <code>state_before</code> contains the state after the update.</p>
<p>The reason is that the values in <code>state_before</code> are <code>Tensor</code>s, which are modified in-place by <code>self.optim.step()</code>.</p>
<p>Maybe you could make a deepcopy of the <code>state_before</code> dict if you need it unchanged after the call to <code>self.optim.step()</code>. See <a href="https://discuss.pytorch.org/t/state-dict-changes-with-the-model/47547">this related question</a>.</p>

andrey1981: <p>Thank you, deepcopy is working</p>
RecursionError: When calling cuda.is_available() later on in an experiment

Connor: <p>Hey,</p>
<p>I have a confusing problem with PyTorch.<br>
I do something with my code which is probably not even advisable - so I will just do a workaround to avoid this problem. Nonetheless, it is puzzling and I wanted to ask if somebody has experienced it as well or knows a solution.</p>
<p>In my experiments, I call cuda.is_available() everytime when I create a new local model, which is superfluous as I realized by now, but sometimes after a bigger amount of training episodes under certain seeds this leads to a “RecursionError: maximum recursion depth exceeded”</p>
<p>The stacktrace always ends as follows</p>
<p><code>File “/home/user/../.venv/lib/python3.12/site-packages/torch/cuda/_</code><em><code>init</code></em><code>_.py”, line 165, in is_available </code><br>
<code>if _nvml_based_avail(): </code><br>
<code>^^^^^^^^^^^^^^^^^^^ </code></p>
<p><code>File “/home/user/../.venv/lib/python3.12/site-packages/torch/cuda/_</code><em><code>init</code></em><code>_.py”, line 158, in _nvml_based_avail </code><br>
<code>return os.getenv(“PYTORCH_NVML_BASED_CUDA_CHECK”) == “1” </code></p>
<p><code>^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^</code><br>
<code>File “”, line 812, in getenv File "</code><br>
<code>&lt;frozen _collections_abc&gt;", line 807, in get</code><br>
<code>File “”, line 711, in _</code><em><code>getitem</code></em><code>_</code><br>
<code>RecursionError: maximum recursion depth exceeded</code></p>
<p>As far as I understand this strack trace, this means that there is a considerate recursion in the os.getenv function calling get and getitem .. This might mean that something wraps getenv in my process and does something weird with it.</p>
<p>Now the puzzling part: My project does not do this. It barely even uses any function from the os module and I’m sure it does not wrap any os functions.</p>
<p>I’m using numpy 2.3.2, pytorch 2.71, tensortdict 0.9.1 (but barely), torchvision 0.22.0 (only for transforms very recently and the bug is older), gymnasium , tqdm, minigrid, vizdoom, scipy, pandas, colorcet 3.1.0.</p>

Connor: <p>Thanks for your response <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">. Yeah, I think I will just follow this recommendation and call torch.cuda.is_available() at the beginning of an experiment and be done with it..</p>
Questions on Model Ensembling Tutorial using vmap

Connor: <p>Hey,</p>
<p>I stumbled on the model ensembling tutorial from the official PyTorch website. It stacks the parameters of the ensemble that share all the same architecture and uses vmap to vectorise the forward propagation. The blog entry promises a significant speedup and it looked quite fitting for what I’m trying since I’m working with something quite similar to an ensemble.</p>
<p><a href="https://pytorch.org/tutorials/intermediate/ensembling.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/tutorials/intermediate/ensembling.html</a></p>
<p>I have a few questions about that tutorial which I found at least not easy to answer for myself.</p>
<p>The forward propagation worked just fine</p>
<p>Consequently, I thought: Wait could you use the same technique for a training loop speeding up that as well?</p>
<p>So, I did the same thing with basically the same input for all models and used the prediction as a prediction in the loss. Each model has its own optimizer. So I zeroed the gradients of each model in a loop. Then I calculated the loss and wanted to calculate the gradient. Afterwards, I iterated over the optimizers and called step.</p>
<p>Admittedly, I probably already messed up for the loss since I didn’t deactivate the reduction… I want each model to have its own loss and being not effected by that of the others.</p>
<p>However, more problematic: The gradient calculation seemed not to work. The models received no gradients and their gradient fields remained None.</p>
<p>So my question is: Does this idea work at all for backpropagation or does it break the computational graph somehow?</p>
<p>Aside of that, the operation is vectorised. Does that mean that PyTorch parallelises the different forward propagations on-the-fly for one GPU? Is it one tensor operation in the end? Or is it still basically syntactic sugar for looping but done in C++?</p>
<p>Thanks for reading <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

vikram2000b: <p>Were you able to find any solutions for this? I am also facing the same issue while trying to speed up my training using vmap.</p>

Connor: <p>Sorry, I haven’t looked into the pytorch forums for.. a while.<br>
I think I know the problem now. (And I bet you probably solved this yourself)</p>
<p>When you use this vmap method to forward through an ensemble of neural networks with the same architecture it disconnects the registering in the optimiser. So, you have to re-register the ensemble then it should work.</p>
<p>For me this was no solution because I wanted to train the single models independently with a loss per member and it is intended to be used with one loss for the whole ensemble.  I think my use-case is not so easy with this method. (At least as far as I understand it)</p>
Prepare imagenet for training

masteryoda: <p>I downloaded the imagenet ILSVRC2012 dataset, but I can’t understand how to prepare it for training.<br>
I have:</p>
<ol>
<li>
<strong>train</strong> folder with 1000 folders (folder per class) with arbitrary name (n01440764, n01443537, …)</li>
<li>
<strong>val</strong> folder with 50,000 JPEG images with arbitrary name (ILSVRC2012_val_00000001.JPEG, ILSVRC2012_val_00000002.JPEG, …)</li>
<li>
<strong>val</strong> labels file - ILSVRC2012_validation_ground_truth.txt</li>
</ol>
<p>Is there any common/easy way to do it with Pytorch?</p>

undefined443: <h2><a name="p-474348-using-imagenet-class-1" class="anchor" href="#p-474348-using-imagenet-class-1"></a>Using ImageNet Class</h2>
<p>If you downloaded the ILSVRC2012 tarball, you can use the <code>torchvision.datasets.ImageNet</code> class.</p>
<p>Directory structure:</p>
<pre><code class="lang-auto">ILSVRC2012
├── ILSVRC2012_img_train.tar
├── ILSVRC2012_img_val.tar
└── ILSVRC2012_devkit_t12.tar.gz
</code></pre>
<p>Code:</p>
<pre data-code-wrap="python"><code class="lang-python">from torchvision.datasets import ImageNet

dataset = ImageNet(root='ILSVRC2012', split="train", transform=None)  # {train|val}
</code></pre>
<p>The ImageNet class will automatically extract the tarball and load the data.</p>
<p>Reference: <a href="https://docs.pytorch.org/vision/stable/generated/torchvision.datasets.ImageNet.html" rel="noopener nofollow ugc">ImageNet | Torchvision documentation</a></p>
<h2><a name="p-474348-using-imagefolder-class-2" class="anchor" href="#p-474348-using-imagefolder-class-2"></a>Using ImageFolder Class</h2>
<p>If you have any directory tree with the ImageNet structure like:</p>
<pre><code class="lang-auto">data
├── class_1
│   ├── img_1_1
│   └── img_1_2
├── class_2
│   ├── img_2_1
│   └── img_2_2
├── ...
└── class_n
    ├── img_n_1
    └── img_n_2
</code></pre>
<p>Then you can use the ImageFolder class to load this directory tree as a dataset:</p>
<pre data-code-wrap="python"><code class="lang-python">from torchvision.datasets import ImageFolder

dataset = ImageFolder(root='data', transform=None)
</code></pre>
<p>Reference: <a href="https://docs.pytorch.org/vision/stable/generated/torchvision.datasets.ImageFolder.html" rel="noopener nofollow ugc">ImageFolder | Torchvision documentation</a></p>
How to properly manage tensor memory when using PyTorch C++ API?

sopelos: <p>Hello</p>
<p>I am experimenting with the PyTorch C++ frontend &amp; noticed some unexpected memory behavior when working with tensors. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=14" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"> Specifically; when I create tensors inside a loop and pass them to functions; I sometimes see memory usage growing continuously, as if the tensors are not being released. <img src="https://discuss.pytorch.org/images/emoji/apple/innocent.png?v=14" title=":innocent:" class="emoji" alt=":innocent:" loading="lazy" width="20" height="20"></p>
<p>This does not happen in my Python experiments; so I suspect I am missing something about how the C++ API handles reference counting and tensor ownership.<img src="https://discuss.pytorch.org/images/emoji/apple/innocent.png?v=14" title=":innocent:" class="emoji" alt=":innocent:" loading="lazy" width="20" height="20"></p>
<p>I tried using <code>at::Tensor::detach()</code> and also checked whether <code>torch::NoGradGuard</code> could help, but the issue persists in long-running processes. <img src="https://discuss.pytorch.org/images/emoji/apple/innocent.png?v=14" title=":innocent:" class="emoji" alt=":innocent:" loading="lazy" width="20" height="20"> It is not always easy to reproduce in small code samples  but in larger projects, the memory growth becomes noticeable after running many iterations. <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=14" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"> Checked <a href="https://docs.pytorch.org/docs/stable/cpp_index.html" class="inline-onebox" rel="noopener nofollow ugc">C++ — PyTorch 2.8 documentation</a> guide for reference. I am also curious if insights from PyTorch C++ memory handling could be applied when learning <a href="https://www.igmguru.com/blog/build-ai-agents-from-scratch" rel="noopener nofollow ugc">How To Build AI Agents</a> efficiently.</p>
<p>I want to understand if this is a misuse of the API on my side / if there are specific cleanup steps required in C++ that are handled automatically in Python.<img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=14" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>
<p>Has anyone else run into this? If so; what is the recommended pattern for managing tensor lifetimes safely in C++? <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=14" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>
<p>Any best practices / examples would be very helpful; since good memory management is crucial for production C++ projects using PyTorch.</p>
<p>Thank you !!<img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=14" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>
Is DMA possible?

alshdavid: <p>Context: <strong>Direct memory access</strong> (<strong>DMA</strong>) is a feature of computer systems that allows certain hardware subsystems to access main system memory independently of the CPU <a href="https://en.m.wikipedia.org/wiki/Direct_memory_access" rel="noopener nofollow ugc">[1]</a></p>
<p>Prelude:</p>
<p>GPUs accessing the system RAM directly rather than depending on GPU memory means larger models can be run on consumer hardware.</p>
<p>Many LLMs require enormous amounts of memory to run, and tasks like video generation are also memory intensive.</p>
<p>While the reduced bandwidth is an obvious performance hurdle, hobbyists/enthusiasts who cannot justify a multi-GPU rig are often okay with taking a performance hit in exchange for the ability to use things.</p>
<p>After all, being able to use something is better than not being able to use something.</p>
<p>Performance, while degraded, will still be significantly greater than CPU training/inference - so it’s a win.</p>
<p>It may also encourage hardware designers to prioritize more efficient pathways for GPU &amp; System RAM (think, Apple’s UMA).</p>
<p>Question:</p>
<p>Does pytorch support DMA APIs or have an abstraction to allow sharing system RAM without changes to client applications</p>
<p>e.g.</p>
<pre data-code-wrap="bash"><code class="lang-bash">env TORCH_GPU_DMA=true python ./comfyui.py
</code></pre>

ptrblck: <p>You can use UVM via a custom <code>MemPool</code> as described <a href="https://docs.pytorch.org/docs/stable/notes/cuda.html#mixing-different-cuda-system-allocators-in-the-same-program">here</a> but would of course run into the performance hit as also explained in the <code>Note</code>.</p>

alshdavid: <p>Is this CUDA specific? Does this also support Vulkan and ROCm?</p>
How to turn off inlining / force materialization in TorchInductor during torch.compile?

skv66: <p>Hi all,</p>
<p>I’m trying to <strong>reproduce the “Without inlining” ablation</strong> from the PyTorch 2.0 paper (the table that reports speedups for “All TorchInductor optimizations”, and then “Without inlining”, “Without fusion”, and “Without fusion &amp; inlining”, etc.). My research goal is to <strong>turn off inlining</strong> <strong>(as well as other optimizations) in TorchInductor</strong> while keeping the rest of the pipeline intact, so I can attribute speedups precisely.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e479597471ac47f7ffac11a6bea8916e7964fa9b.png" data-download-href="https://discuss.pytorch.org/uploads/default/e479597471ac47f7ffac11a6bea8916e7964fa9b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e479597471ac47f7ffac11a6bea8916e7964fa9b.png" alt="image" data-base62-sha1="wBaQTN4Vd3xp4OtBQUwmfEFg4A3" width="285" height="141"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">571×283 54.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Questions</strong></p>
<ol>
<li><strong>Is there an official flag or env var to disable inlining</strong> (i.e., force materialization) in TorchInductor? If so, what is its name and recommended usage?</li>
<li><strong>How do I verify</strong> inlining is off?</li>
</ol>
<p>Thanks!</p>
RTX 5090 Not Detected by PyTorch 2.2.0

Pete_Broznak: <p>PyTorch 2.2.0+cu121 fails to detect NVIDIA GeForce RTX 5090 (32GB) even though the GPU is properly installed and recognized by nvidia-smi.</p>
<h3><a name="p-474311-environment-1" class="anchor" href="#p-474311-environment-1"></a><strong>Environment</strong></h3>
<ul>
<li>
<p><strong>PyTorch Version</strong>: 2.2.0+cu121</p>
</li>
<li>
<p><strong>GPU</strong>: NVIDIA GeForce RTX 5090 (32GB GDDR7)</p>
</li>
<li>
<p><strong>CUDA Version</strong>: 12.8 (Driver 570.181)</p>
</li>
<li>
<p><strong>OS</strong>: Ubuntu (Linux)</p>
</li>
<li>
<p><strong>Python</strong>: 3.12.3</p>
</li>
<li>
<p><strong>Motherboard</strong>: X870E</p>
</li>
</ul>
<h3><a name="p-474311-reproduction-2" class="anchor" href="#p-474311-reproduction-2"></a><strong>Reproduction</strong></h3>
<pre><code class="lang-auto">import torch
print(f"CUDA Available: {torch.cuda.is_available()}")  # Returns False
print(f"GPU Count: {torch.cuda.device_count()}")       # Returns 0
</code></pre>

ptrblck: <p>This is expected since Blackwell support landed in CUDA 12.8+. Install any of our nightly or stable releases built with CUDA 12.8+ and it’ll work.</p>

Pete_Broznak: <p>RTX 5090 Working with PyTorch!</p>
<p>PyTorch nightly build with CUDA 12.8 support exists! Tough find and thank you</p>
<pre><code class="lang-auto">pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu128
</code></pre>
CUPTI Counter Analysis

tohskai: <p>Is there a way to enable CUPTI metrics in kineto like described in HTA? Maybe with some changes to custom build?</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://hta.readthedocs.io/en/latest/source/features/cupti_counter_analysis.html">
  <header class="source">

      <a href="https://hta.readthedocs.io/en/latest/source/features/cupti_counter_analysis.html" target="_blank" rel="noopener nofollow ugc">hta.readthedocs.io</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://hta.readthedocs.io/en/latest/source/features/cupti_counter_analysis.html" target="_blank" rel="noopener nofollow ugc">CUPTI Counter Analysis — Holistic Trace Analysis 0.5.0 documentation</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Thank you for your help!</p>
Package to interactively visualize large Pytorch models in notebooks - torchvista

dev-table: <p>Hi,<br>
I want to share a new open source package I’ve created to interactively visualize Pytorch models in notebooks (like Jupyter, Google Colab etc). It is called torchvista.</p>
<p><strong><a href="https://github.com/sachinhosmani/torchvista" rel="noopener nofollow ugc">Github page</a></strong></p>
<p>With one line of code, you can trace and visualize the forward pass of your Pytorch model.</p>
<p><img src="https://towardsdatascience.com/wp-content/uploads/2025/07/demo8.gif" alt="torchvista in action" width="690" height="366"></p>
<p>It gives you an interactive graph of the model showing how tensors are passed, and lets you explore very large models through expanding/collapsing submodules of interest.</p>
<p>Here are some web-based demos of real-world models if you want check them out before trying torchvista yourself: <a href="https://sachinhosmani.github.io/torchvista/" rel="noopener nofollow ugc">Demos page</a><br>
You can also find a Google Colab tutorial on the Github page.</p>
<p>I hope this is useful, and am curious to get your feedback!</p>
<p>Thank you</p>

jhonsip: <p>How did you create the graph?</p>

dev-table: <p>It had to be done by temporarily wrapping the inbuilt operations and modules to do the additional work of building the graph when they get called. Connections are made from each node to the other nodes it receives tensors from.</p>
<p>When the tracing is done, the operations and modules are reset back to their original definitions.</p>
Make Image Classification faster

david_pom: <p>In tensorflow i have seen that if we use train_ds = train_ds.prefetch(buffer_size=tf.data.AUTOTUNE) the training for each epoch is significantly faster. i want to know the similar approach for pytorch I have used num_workers, pin_memory and torch.cuda.amp and observed that the training time decreases still it is not comparable to tensorflow. But I prefer pytorch results. So any suggestion to improve the training time?</p>

ptrblck: <p>The <a href="https://docs.pytorch.org/tutorials/recipes/recipes/tuning_guide.html">Performance Tuning Guide</a> could be a good resource to optimize your code further.</p>
Some questions about torch.compile

wyzqwq: <p>Hi Pytorch team,</p>
<p>The article <a href="https://pytorch.org/blog/pytorch-vllm-%E2%99%A5%EF%B8%8F/" rel="noopener nofollow ugc">PyTorch + vLLM = <img src="https://discuss.pytorch.org/images/emoji/apple/heart.png?v=14" title=":heart:" class="emoji" alt=":heart:" loading="lazy" width="20" height="20"> – PyTorch</a> shows <strong>1.05x–1.9x speedups with <code>torch.compile</code></strong> for models like Llama4/Qwen3/Gemma3. To better understand:</p>
<ol>
<li>What <strong>benchmark setup</strong> (hardware, input shape) was used?</li>
<li>Was this measured by comparing <strong>vLLM with vs. without <code>torch.compile</code> enabled</strong>?</li>
<li>Could you share a <strong>reference script/command</strong> for reproducing these tests?</li>
</ol>
<p>Thanks for your insights! This will greatly help the community validate the optimizations.</p>
Using DistributedDataParallel with dataloader num_workers > 0

gtbinf22: <p>Due to the setup of my Dataset class and the size of the data, I need to implement num_workers &gt; 0 for the data loading to run efficiently while training.</p>
<p>But I want to further speed up training. I also have 4 Tesla V100 GPUs available. I tried to implement DistributedDataParallel with num_workers &gt; 0 for the dataloader, but it caused my virtual machine to crash.</p>
<p>Is there any suggested way to do this? If not, how can I speed up data loading AND model training simultaneously? My code is running on 24 CPUs (for data loading) and 1 GPU (for training), with 5 seconds per epoch. Ideally, I’d like to try to speed it up by 4-5x.</p>
<p>Thanks in advance.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <p>Yes, multiple workers in <code>DataLoader</code>s and DDP are compatible and commonly used. I would recommend to try to debug the issue you are seeing if <code>num_workers&gt;0</code> is set and maybe try to run a few reference codes to further isolate the issue. E.g. the official <a href="https://github.com/pytorch/examples/blob/f82f5626b6432b8d0b08d58cc91f3bdbb355a772/imagenet/main.py">ImageNet example</a> can use <code>DDP</code> and multiple workers and might be a good baseline.</p>

MSH4R1F: <p>I found adding the parameter</p>
<p><code>multiprocessing_context='fork'</code> inside my DataLoader initialization to work for me</p>
Performance of repeated torch::jit::load() calls

dshin: <p>Consider:</p>
<pre><code class="lang-auto">void loop() {
  torch::jit::script::Module module;
  while (true) {
    module = torch::jit::load(x);
    ...
  }
}
</code></pre>
<p>If I know that on each <code>load()</code> after the first, that the module architecture will remain unchanged, and that only the weights will change, is it possible to make those <code>load()</code> calls faster somehow?</p>

dshin: <p>The answer to this question is yes:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://docs.pytorch.org/TensorRT/tutorials/_rendered_examples/dynamo/refit_engine_example.html">
  <header class="source">

      <a href="https://docs.pytorch.org/TensorRT/tutorials/_rendered_examples/dynamo/refit_engine_example.html" target="_blank" rel="noopener nofollow ugc">docs.pytorch.org</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://docs.pytorch.org/TensorRT/tutorials/_rendered_examples/dynamo/refit_engine_example.html" target="_blank" rel="noopener nofollow ugc">Refitting Torch-TensorRT Programs with New Weights — Torch-TensorRT...</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Avoidable error due to multiple links to libiomp5md.dll (during runtime by numpy and torch)

LeviMadden: <p>Hi All,</p>
<p>I’ve encountered an error when pytorch and numpy are both imported and use numpy to do anything matrix multiplication based, for example covariance calculation. My current versions are:</p>
<ul>
<li>Python: 3.13.5,</li>
<li>torch: 2.8.0+cu126</li>
<li>numpy: 2.1.3</li>
</ul>
<p>The following code:</p>
<pre data-code-wrap="python"><code class="lang-python">import numpy as np
import torch
np.cov(np.array([[0, 2], [1, 1], [2, 0]]).T)
</code></pre>
<p>results in the error:</p>
<p>“OMP: Error <span class="hashtag-raw">#15:</span> Initializing libiomp5md.dll, but found libiomp5md.dll already initialized.<br>
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see <a href="http://www.intel.com/software/products/support/.%E2%80%9D" rel="noopener nofollow ugc">http://www.intel.com/software/products/support/.”</a></p>
<p>Interestingly, the following code doesn’t throw an error:</p>
<pre data-code-wrap="python"><code class="lang-python">import numpy as np
np.cov(np.array([[0, 2], [1, 1], [2, 0]]).T)
import torch
np.cov(np.array([[0, 2], [1, 1], [2, 0]]).T)
</code></pre>
<p>I had a look into the topics and issues on github and I couldn’t find any issues or solutions, so I’m unsure whether the cause this has been identified before and thought this might be useful to others. Should this error be elevated to an issue on torch’s github?</p>
<p>To any others that encounter this failure mode, I hope the second code block is helpful - no other suggested solutions to the OMP: Error <span class="hashtag-raw">#15</span> have worked for me (can’t post these links due to new user restrictions).</p>
Blace.ai - new C++ Inference SDK & Model Hub

spiegelball: <p>Hey,</p>
<p>I’ve been working with C++ and LibTorch for a while, and kept running into the same recurring issues — figuring out the correct build for each OS, switching between CUDA, MPS, and CPU backends, and dealing with GPU-related crashes that could take down the entire application. We decided to build a project to simplify this process and make deployment more reliable.</p>
<p>It’s called blace.ai. Think of it as a wrapper around LibTorch + ONNX Runtime with:</p>
<ul>
<li>
<p>Same API across Windows, Linux, macOS (Intel + Apple Silicon)</p>
</li>
<li>
<p>TorchScript CPU/CUDA/MPS, ONNX CPU, DirectML all supported without touching the model code</p>
</li>
<li>
<p>Process isolation and serialization of AI workloads, so crashes/leaks don’t kill your main app</p>
</li>
<li>
<p>No Python anywhere, all deps bundled</p>
</li>
<li>
<p><strong>Computation graphs:</strong> Chain multiple models/ops together (e.g., run a detector → segmenter → postprocess) with auto tensor caching so you don’t waste time re-running parts of the pipeline</p>
</li>
<li>
<p>A small but growing “model hub” with pre-optimized TorchScript/ONNX models you can drop in — we’re actively expanding the hub and optimizing models for real-world deployment</p>
</li>
</ul>
<p>Here’s a <a href="https://blace-ai.github.io/blace-ai/getting_started.html" rel="noopener nofollow ugc">quickstart example</a> if you’re curious.</p>
<p>Repo: <a href="https://github.com/blace-ai/blace-ai" rel="noopener nofollow ugc">GitHub</a> | <a href="https://www.blace.ai/hub/" rel="noopener nofollow ugc">Model Hub</a></p>
Bug? RMSNorm slower than LayerNorm

JosAr: <p>Hi,</p>
<p>I have replaced nn.LayerNorm with nn.RMSNorm. RMSNorm should be significantly faster I thought.</p>
<p>In practice this slowed my project down. Now I have written a short script to test, and indeed LayerNorm seems to be much faster. How is this possible? I have entered my specific shapes from my project and the results from the comparison below.</p>
<p>Script to test:</p>
<pre><code class="lang-auto">import torch
import time

batch_size = 1
seq_len = 14400#57600
hidden_dim = 384#192
device = "cuda"  # or "cpu"

# Create sample tensors matching your data
x = torch.randn(batch_size, seq_len, hidden_dim, device=device)

# Initialize both norms
layer_norm = torch.nn.LayerNorm(hidden_dim).to(device)
rms_norm = torch.nn.RMSNorm(hidden_dim).to(device)  # or your implementation

# Warmup
for _ in range(10):
    _ = layer_norm(x)
    _ = rms_norm(x)

# Benchmark LayerNorm
torch.cuda.synchronize()
start = time.time()
for _ in range(1000):
    out_ln = layer_norm(x)
torch.cuda.synchronize()
ln_time = time.time() - start

# Benchmark RMSNorm
torch.cuda.synchronize()
start = time.time()

for _ in range(1000):
    out_rms = rms_norm(x)
torch.cuda.synchronize()
rms_time = time.time() - start

print(torch.__version__)
print(f"LayerNorm: {ln_time:.4f}s")
print(f"RMSNorm: {rms_time:.4f}s")
print(f"RMSNorm is {ln_time/rms_time:.2f}x {'faster' if rms_time &lt; ln_time else 'slower'}")
</code></pre>
<p>And the results are:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Shape</th>
<th>LayerNorm</th>
<th>RMSNorm</th>
</tr>
</thead>
<tbody>
<tr>
<td>(1,14400, 384)</td>
<td>0.0683s</td>
<td>0.1687s</td>
</tr>
<tr>
<td>(1, 57600, 192)</td>
<td>0.2244s</td>
<td>0.3555s</td>
</tr>
</tbody>
</table>
</div><p>shouldn’t the results be exactly the opposite?<br>
I am using torch 2.6.0+cu124; but have similar results with torch 2.8. I am running this on a A100</p>

ptrblck: <p><a href="https://github.com/pytorch/pytorch/pull/153666">This PR</a> should improve it.</p>

JosAr: <p>Indeed; I got it working! Thanks a lot</p>
Installation via Conda

Aditya_Kalluri: <p>I’m trying to install pytorch via conda for CUDA 12.8 but there’s only 3 options being shown here for windows: pip, libtorch and source. What can I look into for reference in this matter</p>

vdw: <p>See official announcement <a href="https://github.com/pytorch/pytorch/issues/138506" rel="noopener nofollow ugc">here</a>.</p>
vLLM + openai/gpt-oss-20b on 3× RTX 3090 (CUDA 12.8) — FlashAttention Error

robin_ali: <p>Hi,</p>
<p>Trying to run <code>openai/gpt-oss-20b</code> on <strong>vLLM</strong> with:</p>
<ul>
<li>
<p>3 × RTX 3090 (24 GB each)</p>
</li>
<li>
<p>CUDA 12.8, Driver 570.xx</p>
</li>
<li>
<p>Python 3, PyTorch 2.7.0</p>
</li>
</ul>
<p>I want to split the model across all GPUs, but I’m getting a <strong>FlashAttention error</strong> (likely needs FA3).</p>
<p>Any tips for multi-GPU vLLM setup and installing FlashAttention 3 for this environment?</p>
<p>Thanks!</p>
Why pytorch is getting killed during training on larger dataset on AWS EC2 instances

frequiem11: <p>I’m kind of new to training models so sorry if it is a blatantly bad question. We are training semantic segmentation model (PidNET) on AWS EC2 instances using pytorch. Our default parameter values for <code>num_worker=8</code> and <code>batch_size=8</code> are given. When our dataset exceeds 10.000 images, with given parameters, pytorch gets killed without any error message. Also AWS Instance gets shutdown so I get to reboot the instance.</p>
<p>Training works for <code>num_worker=4</code> and <code>batch_size=4</code> values when our datasets contain 10.000-14.000 images. However, when our datasets exceeds 14.000 images, the python script gets killed again without no errors or warnings whatsoever.</p>
<p>We have plenty of storage so it should not be an issue. GPU Memory is also sufficient as I check with <code>nvidia-smi -l 10</code> command. What could be the issue?</p>

PyDevC: <p>what about RAM?</p>
<p>How much size is the image dataset?</p>

spiegelball: <p>Are you using docker? If so, check <code>shm_size</code> in <a href="https://docs.gitlab.com/runner/configuration/advanced-configuration/" class="inline-onebox" rel="noopener nofollow ugc">Advanced configuration | GitLab Docs</a></p>

frequiem11: <p>After check, It gets killed when RAM is full.</p>

frequiem11: <p>Yes I am using docker. After checking out, when the system RAM is full, it gets killed.</p>

spiegelball: <p>Did you set the variable in the runner?</p>
<pre><code class="lang-auto">[runners.docker]
    image = "ruby:3.1"
    gpus = "all"
    shm_size = 2073741824
...
</code></pre>
<p>It needs a pretty high value.</p>

PyDevC: <p>yes I also encountered this when trying to run <a href="https://github.com/lllyasviel/FramePack.git" class="inline-onebox" rel="noopener nofollow ugc">GitHub - lllyasviel/FramePack: Lets make video diffusion practical!</a></p>
<p>well I could not found a solution other than buy more ram <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=14" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>
Help getting a project that requires torch==2.6.0 torchvision==0.21.0 on Blackwell

mikecaronna: <p>Hello. I’d really appreaciate any advice on how to resolve this.</p>
<p>I’m trying to get a project running on Ubuntu Linux ( <a href="https://github.com/nv-tlabs/GEN3C/blob/main/requirements.txt" class="inline-onebox" rel="noopener nofollow ugc">GEN3C/requirements.txt at main · nv-tlabs/GEN3C · GitHub</a> ) but encountering errors on my RTX5000 series GPU. The project requires Python 3.1, torch 2.6.0, and torchvision 0.21.0. The error I’m getting is:</p>
<p>…CUDA capability sm_120 is not compatible with the current PyTorch installation. The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_70 sm_75 sm_80 sm_86 sm_90.</p>
<p>I’ve looked in the compatibility Matrix but I don’t see this combination. I’m assuming that I need to compile torch and torchvision  with the required versions AND include CUDA 12.8 or 12.9 compatibility, but I don’t know how to do this.</p>
<p>If anyone could point me to a guide or in the general right direction, I’d appreciate it. Thanks!</p>
<p>Nvidia-smi reports:<br>
±----------------------------------------------------------------------------------------+<br>
| NVIDIA-SMI 577.00                 Driver Version: 577.00         CUDA Version: 12.9     |<br>
|-----------------------------------------±-----------------------±---------------------+<br>
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |<br>
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |<br>
|                                         |                        |               MIG M. |<br>
|=========================================+========================+======================|<br>
|   0  NVIDIA RTX PRO 6000 Blac…  WDDM  |   00000000:08:00.0  On |                  Off |<br>
| 30%   34C    P8             23W /  600W |    1356MiB /  97887MiB |      0%      Default |<br>
|                                         |                        |                  N/A |<br>
±----------------------------------------±-----------------------±---------------------+</p>

ptrblck: <p>I would recommend checking why your project requires <code>torch==2.6.0</code> and relax this version pin if possible allowing you to use newer PyTorch releases which already support Blackwell GPUs. I’m not aware of any breaking changes between 2.6 and 2.7 where Blackwell support was introduced.</p>
In multi-processing, when one process exits unexpectedly, how to get others out of hang?

Fang: <p>Suppose I have 4 processes. With the current torchrun implementation, if one process exits unexpectedly, the other three will automatically be closed. However, I’m looking to modify this behavior so that the remaining three processes can stay alive to implement fault tolerance.</p>
<p>A key issue I’m facing is that these three remaining processes end up hanging while waiting to communicate with the exited process. Is there a way to get them out of this state?</p>
<p>I’ve attempted to call torch.distributed.distributed_c10d.ProcessGroupNCCL.abort() from a separate watchdog thread in each of the three remaining processes, but this approach hasn’t worked. Instead, the watchdog thread itself gets stuck in the abort() function. Is this abort() method supposed to be called by all initial processes?</p>
What are liquid neural networks?

Aqib_Waqar: <p>Artificial intelligence is evolving rapidly, and one of the most fascinating advancements is the emergence of Liquid Neural Networks. Unlike traditional models, these networks can adapt to new information in real-time, making them highly efficient and dynamic. This means AI can perform better in unpredictable environments and continue learning long after deployment. If you’re curious to understand how this groundbreaking technology works and why experts believe it could transform the future of AI, read the full guide here: <a href="https://duhmagazine.com/what-are-liquid-neural-networks/" rel="noopener nofollow ugc">What Are Liquid Neural Networks</a>.</p>

MYEV: <p><strong>Liquid Neural Networks (LNNs)</strong> are a new type of artificial neural network designed to adapt their internal parameters continuously over time, making them more flexible and better suited for dynamic, real-world environments compared to traditional static networks.</p>
<p>Here’s what makes them special:</p>
<ul>
<li>
<p><strong>Dynamic behavior:</strong><br>
Unlike conventional neural networks where weights are fixed after training, LNNs can change their internal connections while processing data<a href="https://www.myev.org.in/" rel="noopener nofollow ugc">,</a> allowing them to respond to new or unexpected inputs in real time.</p>
</li>
<li>
<p><strong>Inspired by biology:</strong><br>
Their design is loosely based on how biological neurons in living organisms adapt to constantly changing surroundings.</p>
</li>
<li>
<p><strong>Small but powerful:</strong><br>
They often require fewer parameters than large deep learning models, yet they can handle complex tasks effectively—especially in scenarios like autonomous driving, robotics, and real-time decision-making.</p>
</li>
<li>
<p><strong>Better generalization:</strong><br>
Because they keep adapting after deployment, LNNs can maintain accuracy even when facing data they’ve never seen before, unlike static models that often degrade outside their training domain.</p>
</li>
</ul>
TrivialAugmentWide on segmentation

David_Hresko: <p>Hi, what is correct process to apply TrivialAugmentWide to both on input and label when I want to perform segmentation task ?</p>

ptrblck: <p>I guess a safe approach would be creating a custom transform module accepting two inputs (the data input and segmentation mask) by either deriving from the transformation directly or by copying the source code. This would allow you to keep the randomness logic in the <code>forward</code> and then to apply the actual transformation op on both tensors.<br>
Re-seeding the code could also work but could be tricky to get right as you would need to guarantee the same calls into the pseudorandom number generator and could have side effects on the actual training.</p>

Introvert_Loner: <p>It might be late, but better late than never, Here’s what i tried</p>
<p>I concatenate the channels together (e.g. Image with shape of (B,3,H,W) and Segments are of (B,1,H,W)  – augmented = torch.cat((image,segments), dim =1). After that you just have to indexing them out, i.e. image = augmented[:,:3] and segments = augmented[:,-1:]. After that, you can visualize to check the results.</p>
<p>Hope it helps.</p>
Cleanup for multiprocess DataLoader

markl: <p>I have a server program, written in C++, that fetches training data. I’ve wrapped the server program with a python Dataset.</p>
<pre data-code-wrap="python"><code class="lang-python">class MyDataset(torch.utils.data.Dataset):
    def __init__(self):
        super().__init__()
        self.server = None
        self.init_server()

    def deinit_server(self):
        # send sigint to the existing server [omitted]
        self.server = None

    def init_server(self):
        self.server = subprocess.Popen(
            [SERVER_PATH, SERVER_ARG1, SERVER_ARG2, ...],
            preexec_fn=os.setsid
        )

    def __getitem__(self, idx):
        # communicate with server [omitted]

    def __len__(self):
        # communicate with server [omitted]

    def __del__(self):
        # send sigint to server [omitted]
</code></pre>
<p>This works fine until I try to use the multiprocessing data loader.</p>
<pre data-code-wrap="python"><code class="lang-python">    dataset = MyDataset()
    dataset.deinit_server() # prevents any weird interaction between multiprocessing.Process and subprocess.Popen

    def worker_init_fn(*args, **kwargs):
        dataset = torch.utils.data.get_worker_info().dataset
        dataset.init_server() # every fork of the dataset has its own server

    data_loader = DataLoader(
        dataset=dataset,
        worker_init_fn=worker_init_fn,
        batch_size=batch_size,
        num_workers=num_workers,
        collate_fn=collate_fn,
        persistent_workers=True,
        shuffle=True
    )
</code></pre>
<p>I use the data loader in the normal way, but when the script exits, all the server subprocesses remain alive. It would be nice for the DataLoader to call <code>__del__</code> for all the forked datasets, so that my server subprocesses can be cleaned up.</p>
<p>I’m not sure how to accomplish this.</p>
"No kernel image is available for execution on this device"

Luis_Park: <p>Hey guys! I currently have a project that is using torch==2.5.1+cu121 and it seems to be working just fine on my rtx3090. I have torch downloaded in a conda env, and have bundled this project into a exe file using pyinstaller. This also seems to be working fine; however when I try running this exe file on a different machine using rtx4080, im getting a “No kernel image is available for execution on this device” error. Was wondering why this was happening</p>
<p><img src="https://mail.google.com/mail/u/1?ui=2&amp;ik=3d0744acca&amp;attid=0.1&amp;permmsgid=msg-a:r4428200993063769872&amp;th=1989efda14041ac3&amp;view=fimg&amp;fur=ip&amp;permmsgid=msg-a:r4428200993063769872&amp;sz=s0-l75-ft&amp;attbid=ANGjdJ8MBNSHv1gwHz6MPUEiLvjcGXtJvkmfXHeTdBAHA31UbWr-xtxphaPD6H3cMcAPzh-IxvxYgXLI7c5rWPz_4CAueRULxXP775cChI-fVfBhbFXp4YqlCukJC7U&amp;disp=emb&amp;realattid=ii_1989efd92dcb08c77e51&amp;zw" alt="" role="presentation" width="" height=""></p>

ptrblck: <p>The RTX 4080 reuses the same kernels as the 30XX series. Make sure the same PyTorch build is used and if so, try to narrow down which kernel fails to check if a 3rd party package is causing the issue.</p>