akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
FP16 support in pytorch mobile

Ambrozy: <p>Is there a way to run scripted half precision model in pytorch mobile? If not, will this be supported in the future? and is there way to at least convert saved fp16 scripted model to fp32 on mobile device inside pytorch mobile?</p>

ptrblck: <p>I’m not sure if the mobile device architecture would benefit from <code>float16</code> dtypes (the x86 CPU architectures do not see a benefit from it which is also the reason <code>float16</code> ops are mostly not supported on the CPU in PyTorch).</p>
<p>I don’t know about future plans and if some devices are planning to support <code>float16</code> on mobile.</p>
<p>I would assume transformations are still possible on mobile, i.e. would <code>model.to(dtype)</code> work?</p>

kimishpatel: <p>While arm cpus have support for fp16, we dont support that on mobile runtime to directly take advantage of fp16 features of mobile cpus.</p>
<p>In terms of saving fp16 and converting to fp32 on model load is something that is not quite supported natively. I have not explored this but it might be possible to do this entirely in torchscirpt without pytorch runtime needing to support this natively.</p>

kimishpatel: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="168990">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I would assume transformations are still possible on mobile, i.e. would <code>model.to(dtype)</code> work?</p>
</blockquote>
</aside>
<p>model.to likely wont work since on server with full pytorch runtime gets torch::jit::Module as the type returned by torch::jit::load whereas on mobile we have torch::jit::_load_for_mobile which returns torch::jit::mobile::Module.</p>

brainchip_india: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I’m trying a FP32 and FP16 PyTorch model on Mobile. FP32 works fine, but with FP16 the app crashes.<br>
Does Pytorch on mobile have support for FP16 now?</p>

ptrblck: <p>Which backend are you using? Do you see these crashes in ExecuTorch?</p>
Basic System requirement to be able to contribute to Pytorch

Bishwa_Karki: <h1><a name="p-476356-hi-1" class="anchor" href="#p-476356-hi-1"></a>Hi,</h1>
<p>I have been using Pytorch since a while and recently I am thinking of contributing to it. I know before making contribution I have to understand the entire project outlet and end to end working. But my goal is to be able to contribute at the very core level.</p>
<p>My question is, if there any current contributors, what is the spec of the system do you use or should suffice for most task?</p>
<p>I was trying to reproduce an error which was a matmul computation of large number and it required 21GB of RAM. And I am curious what system does the contributors use ? Or if most have access to servers from labs ? Currently, I am individual contributors doesn’t have access to any HPC.</p>
<p>Currently I am using Mac M2, 64 bit, 8GB RAM and this is for sure not sufficient may be even to compile and build torch in my local machine lol.</p>
<p>Thanks</p>
MultiheadAttention with multiple batch dimensions

Wei_Min_Loh: <p>For my use-case, the input dimensions are (N, M, L, E), similar to the expected format in <code>nn.MultiheadAttention</code> but with an additional “batch” dimension M. This 4D tensor is unfortunately not accepted by <code>nn.MultiheadAttention</code>.</p>
<p>How can I implement self/cross attention to be over the second last dimension in a 4D tensor? I’m hoping for an interface similar to <code>nn.Linear</code> which accepts (*, E) dimension tensors.</p>
<p>Ideally, the solution shouldn’t involve flattening the N and M dimension into a single dimension as the output must be in a 4D tensor format, unless there is an efficient way to reconstruct the 4D structure from 3D with the appropriate attention masks.</p>

valerian.rey: <p>Would it work for you to just vmap the model’s forward pass?</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch import nn, vmap

N, M, L, E = 6, 4, 20, 8
model = nn.MultiheadAttention(embed_dim=E, num_heads=2, batch_first=True)

query = torch.randn((N, M, L, E))
key = torch.randn((N, M, L, E))
value = torch.randn((N, M, L, E))

attn_output, attn_output_weights = vmap(model)(query, key, value)

print(attn_output.shape)
print(attn_output_weights.shape)
</code></pre>
<p>Output:</p>
<pre><code class="lang-auto">torch.Size([6, 4, 20, 8])
torch.Size([6, 4, 20, 20])
</code></pre>

Wei_Min_Loh: <p>Yes, it works great! I read up on <code>torch.vmap</code>, and it seems to leverage GPU parallelism with full gradient support. Thank you!</p>
What version of Pytorch is compatible with NVIDIA GeForce GTX 1080

00krishna: <p>I updated my Pytorch installation last night, but now I am getting a message:</p>
<blockquote>
<p>UserWarning:<br>
Found GPU1 NVIDIA GeForce GTX 1080 which is of cuda capability 6.1.<br>
PyTorch no longer supports this GPU because it is too old.<br>
The minimum cuda capability supported by this library is 7.5.</p>
</blockquote>
<p>My current Cuda version is 12.9. I am running Ubuntu 24.04.</p>
<p>I checked Chatgpt and it was telling me all sorts of things about using using CUDA 11.8 and reducing the versions for a bunch of different packages. That sounded a bit suspicious to me, so I figured I would check.</p>
<p>I consulted the documentation on the pytorch website, but all of the different versions of cuda, and pytorch, and cudatoolkit, etc., are confusing me. And I am worried about doing major surgery on my cuda installations and then putting that whole ecosystem into an unrecoverable state.</p>
<p>Can someone indicate what the correct pytorch version is for a GTX 1080 now? I imagine pytorch 2.5 works with cuda 12.9, as that is what I had before. Sorry if this question is too direct, however I did consult multiple sources before finally posting here. Any suggestions are appreciated.</p>

ptrblck: <p>Install either any older binaries (&lt;=2.7.1) or our current binaries built with CUDA 12.6.</p>

00krishna: <p>Oh yes, this is exactly what I was looking for. Thanks so much <a class="mention" href="/u/ptrblck">@ptrblck</a> . I get it.</p>
<p>So I just have to install the <code>nvidia-cuda-toolkit</code> that corresponds to CUDA 12.6. Meaning I unintall 12.9 and then install 12.6. Then install pytorch 2.7, and then things should work.</p>
<p>Let me know if I captured anything incorrectly. Otherwise I will proceed down that path. Thanks again for setting things straight.</p>

ptrblck: <p>You don’t need to install a CUDA toolkit yourself unless you want to build PyTorch from source or a custom CUDA extension (or any other CUDA application of course). The PyTorch binaries ship with their own CUDA runtime dependencies and you would only need to properly install an NVIDIA driver from the same major release.</p>

DaAlb: <p>Did it work for you?</p>
<p>2.7.1+ cuda 12.6</p>
<p>I can’t get my 1080ti to work with the setup</p>

valerian.rey: <p>I’m using a GeForce GTX 1080 on ubuntu 22.04 with torch==2.9.0+cu126, and driver version 580.95.05 and it works perfectly. Note that what matters is not the version of the cuda installed on your system (if any) but the version of the cuda that comes with your installation of torch.<br>
To install torch with cuda 12.6:</p>
<pre><code class="lang-auto">pip install torch --index-url https://download.pytorch.org/whl/cu126
</code></pre>
<p>I don’t know about the 1080ti though.</p>
Varying batch size for pre-trained model changes inference result, even in evaluation mode

yatagarasu50469: <p>Well, here’s hoping this behavior is something obvious…Attempting to extract some features using a pre-trained ResNet50 model, which despite being in evaluation mode, is producing different outputs for the same input, if submitted in a larger batch. Tried a couple different versions 2.5.1 and 2.9.0, came up with a minimum reproducible code, switched to AlexNet to avoid potential issues with batch normalization layers, and dropped GPU-acceleration in case of CUDA derived shenanigans. Any insight into how to fix and why result and result2 are not identical would be greatly appreciated!</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torchvision import models

#Setup deterministic behavior
torch.use_deterministic_algorithms(True)

#Set torch seed for consistency
torch.manual_seed(0)

#Create batch of random input data
batch = torch.randn(32, 3, 224, 224)

#Load model and set to evaluation mode
model = models.alexnet(weights='DEFAULT')
model.eval()

#Obtain result for first image, inferencing only it
with torch.no_grad(): result = model(torch.unsqueeze(batch[0], 0)).detach()[0]

#Obtain result for first image, inferencing the whole batch
with torch.no_grad(): result2 = model(batch).detach()[0]

#Results should be identical when model is in eval mode...?
print('Sum Absolute Difference: ', torch.sum(torch.abs(result-result2)).item())

</code></pre>

ptrblck: <p>Small numerical errors caused by the limited floating point precision are expected as these depend on the used algorithm and order of operations used to compute the results. Bitwise identical results are achieved for the same setups which includes the workload with its input shapes when deterministic mode is enabled.</p>
nn.MultiheadAttention fast_path substantially less efficient

robertdvdk: <p>Edited after further debugging and creating an even more minimal example.</p>
<p>Using nn.MultiheadAttention in eval() mode is substantially less efficient than using it in train() mode: in my example, the model uses 2.4GB VRAM in train(), and 10.6GB VRAM in eval(). The issue seems to be the fast_path in nn.MultiheadAttention.forward(), because when I remove the fast_path, the issue is solved.</p>
<p>Minimal example:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn
import torch.nn.functional as F
import gc

def test_mha_directly(mode: str):
    """Test MultiheadAttention directly."""
    device = "cuda"
    torch.cuda.empty_cache()
    torch.cuda.reset_peak_memory_stats()
    gc.collect()

    # torch.backends.mha.set_fastpath_enabled(False)   # UNCOMMENTING THIS LINE SOLVES THE ISSUE
    
    # Create MHA module
    mha = nn.MultiheadAttention(
        embed_dim=256,
        num_heads=4,
        dropout=0,
        batch_first=True,
    ).to(device)
    
    if mode == "train":
        mha.train()
    else:
        mha.eval()
    
    batch_size = 32
    seq_len = 2000
    
    # Create inputs
    query = torch.randn(batch_size, seq_len, 256, device=device)
    key = query
    value = query
    key_padding_mask = torch.zeros(batch_size, seq_len, device=device, dtype=torch.bool)
    attn_mask = torch.zeros(seq_len, seq_len, device=device, dtype=torch.bool)
    
    print(f"Mode: {mode} (single MHA)")
    with torch.no_grad():
        output, _ = mha(query, key, value, 
                    key_padding_mask=key_padding_mask,
                    attn_mask=attn_mask,
                    need_weights=False)
    
    torch.cuda.synchronize()
    peak = torch.cuda.max_memory_allocated() / 1024**2
    print(f"Peak memory: {peak:.2f} MB")
    
    del mha, output
    torch.cuda.empty_cache()
    gc.collect()
    
    return peak

if __name__ == "__main__":
    print("Testing with hooks and direct MHA\n")

    print("\n" + "=" * 60)
    print("Single MultiheadAttention")
    print("=" * 60)
    train_mha = test_mha_directly("train")
    eval_mha = test_mha_directly("eval")
    
    print("\n" + "=" * 60)
    print("SUMMARY")
    print("=" * 60)
    print(f"Single MHA - Train:        {train_mha:.2f} MB")
    print(f"Single MHA - Eval:         {eval_mha:.2f} MB")

</code></pre>
<p>Running on an RTX 4090 with CUDA 12.7. Tried Python 3.12 + PyTorch 2.8.0 and Python 3.14 + PyTorch 2.9.0. Also tried running on a Google Colab instance. With all three setups, the issue persists.</p>
<p>It’s not clear to me why the fast_path would be so much less efficient than the “regular” path. It’s not just less memory efficient, it’s also much slower.</p>
Build Pytorch 2.7.1 from source with cuda 12.4

Shuggurath: <p>I am currently working on an HPC which allows me to install only cuda 12.2 and cuda 12.4 via spack system OS Red Hat Enterprise Linux 8.7 (GCC 12.2, glibc 2.28). I was wondering if there is a way to build from source Pytorch 2.7.1 with CUDA12.4. Unfortunately I can’t use conda so I have to rely on pip or building from source.</p>
<p>Thanks in advance for the help.</p>

ptrblck: <aside class="quote no-group" data-username="Shuggurath" data-post="1" data-topic="223681">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shuggurath/48/78486_2.png" class="avatar"> Shuggurath:</div>
<blockquote>
<p>working on an HPC which allows me to install only cuda 12.2 and cuda 12.4 via spack system OS Red Hat Enterprise Linux 8.7 (GCC 12.2, glibc 2.28).</p>
</blockquote>
</aside>
<p>I’m unsure if I understand your issue correctly, but please note that your locally installed CUDA toolkit won’t be used if you install our PyTorch binaries as they ship with their own CUDA runtime dependencies. You need to properly install an NVIDIA driver and can then install any of our PyTorch binaries which support your GPUs.</p>

Shuggurath: <p>I am currently using a CUDA12.4 compiler installed via spack, I’m working on a node with 4 nvidia A100-SXM-64GB,</p>
<p>driver info:<br>
NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2</p>
Support for NVIDIA RTX 5090 (CUDA SM_120)

bartwojtas: <p>Hi,<br>
I’ve recently tested PyTorch (nightly build with CUDA 12.1) on an NVIDIA RTX 5090 and received a warning:<br>
“sm_120 is not supported”</p>
<p>Will support for sm_120 be included in upcoming builds?</p>
<p>GPU: RTX 5090</p>
<p>Driver: 581.42</p>
<p>CUDA: 12.2</p>
<p>One more thing!</p>
<p>I’m using a new NVIDIA RTX 5090 GPU (compute capability sm_120, CUDA 12.2).<br>
When running YOLOv8 inference on GPU, I get this warning and it fails:</p>
<blockquote>
<p>NVIDIA GeForce RTX 5090 with CUDA capability sm_120 is not compatible with the current PyTorch installation.</p>
</blockquote>
<p>However, YOLOv5 runs on the same setup without issue.</p>

ptrblck: <aside class="quote no-group" data-username="bartwojtas" data-post="1" data-topic="223683">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/258eb7/48.png" class="avatar"> bartwojtas:</div>
<blockquote>
<p>I’ve recently tested PyTorch (nightly build with CUDA 12.1)</p>
</blockquote>
</aside>
<p>PyTorch was built with CUDA 12.1 from our 2.1 to 2.5 release, which are too old too support the latst hardware generation. Blackwell requires CUDA 12.8+ so install any of our binaries built with CUDA 12.8+ instead.</p>
NVIDIA DGX Spark support

f3ba: <p>Does PyTorch support the newly released NVIDIA DGX Spark ?</p>

ptrblck: <p>Yes our latest stable and nightly binaries support DGCSpark. You can install them via <code>pip3 install torch torchvision --index-url ``https://download.pytorch.org/whl/cu130``.</code></p>

f3ba: <p>Can I install with <code>uv</code>?</p>

ptrblck: <p>I believe so but I have not tried it out. Let me know if it works.</p>
Context Parallel – attention mask not supported yet? Performance drop in FSDP2 + CP fine-tuning

HeegyuKim: <p>Hi,</p>
<p>While fine-tuning a sliding-window transformer using <strong>FSDP2 + context_parallel using huggingface accelerate</strong>,<br>
I noticed that training speed improved significantly — but model performance dropped.<br>
After checking the source (<a href="https://github.com/pytorch/pytorch/blob/0fabc3ba44823f257e70ce397d989c8de5e362c1/torch/distributed/tensor/experimental/_attention.py#L688" rel="noopener nofollow ugc">_attention.py#L688</a>),<br>
it seems that <strong>attention operations are not yet supported for context parallel</strong>.</p>
<p>I suspect that the performance degradation is caused by the <strong>attention mask not being applied</strong> during CP execution.</p>
<p>Could you share whether:</p>
<ol>
<li>
<p>attention mask support for context_parallel is planned, and</p>
</li>
<li>
<p>there’s any ETA or related PR for this feature?</p>
</li>
</ol>
<p>Thanks!</p>
PyTorch AD with non-python functions

Chanpisey: <p>I am working on a project that requires using a Torch optimizer with Julia functions. I need to know if Torch’s autograd can compute gradients for functions defined in Julia, or if it is restricted to functions written in Python. Thanks so much!</p>

ptrblck: <p>Custom <code>autograd.Function</code>s should work as these allow you to use any library you want but require the manual implementation of the <code>backward</code> call.</p>
Training a VAE on the embeddings of other VAEs

aajanqd: <p>I’m training a VAE (let’s call this VAE_combined) on the embeddings of three other VAEs (let’s call them VAE_1, VAE_2, and VAE_3). I’m wondering whether I should generate the training dataset for the VAE_combined using VAE_1/2/3.train() or VAE_1/2/3.eval()? Both the test and train losses for VAE_combined when training on embeddings generated using .train() are significantly lower than the losses obtained on embeddings generated using .eval(). But I’m not sure which one is “correct”.</p>

J_Johnson: <p>You’re spot-on about freezing the base VAEs to keep their domain-specific latents. Use .eval() and requires_grad_(False) to ensure they’re fixed:</p>
<pre data-code-wrap="python"><code class="lang-python">for vae in [base_vae1, base_vae2, base_vae3]:
    vae.eval()
    for param in vae.parameters():
        param.requires_grad_(False)
</code></pre>
<p>A few tips to avoid pitfalls:</p>
<ol>
<li>Normalize Latents: Z-scale base VAE latents (subtract mean, divide by std) if their scales differ:</li>
</ol>
<pre data-code-wrap="python"><code class="lang-python">z1 = (z1 - z1.mean()) / z1.std()
z_input = torch.cat([z1, z2, z3], dim=-1)
</code></pre>
<ol start="2">
<li>Use MSELoss for reconstruction (input vs. output latents) plus KL divergence. Consider β-VAE (e.g., β=0.1 initially) to balance KL:</li>
</ol>
<pre data-code-wrap="python"><code class="lang-python">recon_loss = torch.nn.MSELoss()(z_recon, z_input)
kl_loss = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
loss = recon_loss + beta * kl_loss
</code></pre>
<ol start="3">
<li>VAEs can be tricky. If loss isn’t converging, try a lower learning rate (e.g., 1e-4), KL annealing (β from 0 to 1), or monitoring for posterior collapse (KL near zero). Plot loss curves before extending training.</li>
<li>Partial Inputs: For inference with fewer latents, pad missing ones with zeros or their mean to match input size.</li>
</ol>

aajanqd: <p>Thanks for your reply!<br>
What’s the exact reason I should generate embeddings using .eval()? As I mentioned, I get significantly better results when generating using .train(), so it’s tempting to use those.</p>

J_Johnson: <p>You said you get better results. But how many epochs are we talking? Typically, VAEs take a very long time to train (like an order of 100x what other types of networks take). Can you show a chart of loss both with the base VAEs set on eval and on train? I’m wondering if you’re training long enough to get proper convergence.</p>

aajanqd: <p>I don’t have any charts, but I’m training on a small dataset of just 100 data points (the ultimate goal is to build a larger dataset with active learning). Using the same dataset over 100 epochs, the reconstruction error is 3x lower using .train(). By that time, the network seems to have converged.</p>
Mac inter want to install torch version 2.6 or higher

muyang24: <p>My computer is equipped with a Mac Intel processor. I want to install torch version 2.6 or higher, but when I tried to install it using the command “pip install torch&gt;=2.6”, I found that my computer only supports up to version 2.2.2. In this situation, how should I solve the problem?</p>

ptrblck: <aside class="quote" data-post="2" data-topic="204546">
  <div class="title">
    <div class="quote-controls"></div>
    <img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <div class="quote-title__text-content">
      <a href="https://discuss.pytorch.org/t/why-no-macosx-x86-64-build-after-torch-2-2-2-cp39-none-macosx-10-9-x86-64-whl/204546/2">Why no macosx x86_64 build after torch-2.2.2-cp39-none-macosx_10_9_x86_64.whl</a> <a class="badge-category__wrapper " href="/c/mac-os-x/32"><span data-category-id="32" style="--category-badge-color: #283890; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category --style-square "><span class="badge-category__name">Mac OS X</span></span></a>
    </div>
  </div>
  <blockquote>
    PyTorch macOS x86 builds were deprecated as explained <a href="https://dev-discuss.pytorch.org/t/pytorch-macos-x86-builds-deprecation-starting-january-2024/1690">here</a> and <a href="https://github.com/pytorch/pytorch/issues/114602">here</a>. torch==2.2.x was the last release for it.
  </blockquote>
</aside>
Error(s) in loading state_dict for ResNet: 	Missing key(s) in state_dict:

walid_ali_aydi: <p>good afternoon,</p>
<p>i tried to implement the source code in the link below. after finishing I tried to load the best_model.pt  model using the following code, but i get the error:Error(s) in loading state_dict for ResNet: 	Missing key(s) in state_dict:</p>
<p>from timm import create_model</p>
<p>import torch</p>
<p>modelgray = create_model(</p>
<pre><code>    "resnetrs50",

    pretrained='True',

    num_classes=3,

    in_chans=1,

)
</code></pre>
<p>state_dict = torch.load(‘best_model-gray.pt’, weights_only=True, map_location=‘cpu’)</p>
<p>modelgray.load_state_dict(state_dict)</p>
<p>this is how i call the main.py in colab:<br>
!python main.py --data_dir /train_test_dataset_path/ --epochs 4 --greyscale 1 --pretrained 1 --freeze 1 --batch-size 32 --num_chans 1</p>
<p>thanks an advance</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://medium.com/data-science/transfer-learning-on-greyscale-images-how-to-fine-tune-pretrained-models-on-black-and-white-9a5150755c7a">
  <header class="source">
      <img src="https://miro.medium.com/v2/5d8de952517e8160e40ef9841c781cdc14a5db313057fa3c3de41c6f5b494b19" class="site-icon" alt="" width="32" height="32">

      <a href="https://medium.com/data-science/transfer-learning-on-greyscale-images-how-to-fine-tune-pretrained-models-on-black-and-white-9a5150755c7a" target="_blank" rel="noopener nofollow ugc" title="08:38AM - 24 February 2022">Medium – 24 Feb 22</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:510/500;"><img src="https://miro.medium.com/v2/resize:fit:940/1*UJ8SZBLWkNVNv6rBRv_1sg.png" class="thumbnail" alt="" width="510" height="500"></div>

<h3><a href="https://medium.com/data-science/transfer-learning-on-greyscale-images-how-to-fine-tune-pretrained-models-on-black-and-white-9a5150755c7a" target="_blank" rel="noopener nofollow ugc">Transfer Learning on Greyscale Images: How to Fine-Tune Pretrained Models on...</a></h3>

  <p>Everything you need to know to understand why the number of channels matters and how to work around this</p>

  <p>
    <span class="label1">Reading time: 19 min read</span>
  </p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
Best environment for deeplearning on a local GPU workstation

dlgmlwndlgml: <p>Hi everyone,</p>
<p>I recently built a local workstation for deep learning research, and I’d like to fully utilize my GPU with PyTorch instead of using Google Colab.</p>
<p><strong>Specs</strong></p>
<ul>
<li>CPU: Intel i9-11900K (8 cores, 11th Gen)</li>
<li>GPU: NVIDIA RTX 5070 (12 GB VRAM)</li>
<li>RAM: 128 GB DDR5</li>
<li>SSD: 2 TB NVMe</li>
<li>OS: Windows 11 / Ubuntu WSL2 (optional)</li>
</ul>
<p><strong>Goal</strong><br>
To train deep learning models (CNNs, Transformers) with PyTorch using full GPU acceleration.</p>
<p><strong>Tools I’m considering</strong></p>
<ul>
<li>JupyterLab</li>
<li>VS Code</li>
<li>(others welcome)</li>
</ul>
<p><strong>Question</strong><br>
Which environment is most suitable for PyTorch GPU development on a local workstation?</p>
<p><em>(I’m a deep learning beginner, but my professor asked me to check this setup, so I’m trying to find the best environment before we start our experiments.)</em></p>
<p>Thanks in advance for any insights or setup recommendations!</p>

vdw: <aside class="quote no-group" data-username="dlgmlwndlgml" data-post="1" data-topic="223659">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dlgmlwndlgml/48/78489_2.png" class="avatar"> dlgmlwndlgml:</div>
<blockquote>
<p><strong>Goal</strong><br>
To train deep learning models (CNNs, Transformers) with PyTorch using full GPU acceleration.</p>
</blockquote>
</aside>
<p>You may want to elaborate on this.</p>
<p>You can train such models on consumer GPUs, but the size of the models will always be limited by the VRAM. If it’s for protoypying or simply for learning, your setup is probably fine. Well, you should definitely go for the 5070 Ti as it comes with 16 GB VRAM.</p>
<p>But again, particularly when you consider training / fine-tuning Transformer-based LLMs, 16 GB VRAM is likely to be very quickly a limiting factor.</p>

J_Johnson: <p>I’m a big fan of PyCharm. If you’re used to Windows interfaces, it’s very intuitive. Plus the integrated code debugger makes finding most oversights in your code a breeze.</p>
No error raised despite shape mismatch in load_state_dict

mh2parker: <p>When I initialize a Parameter of a Module to be a scalar, and then load in a state_dict containing a 1D tensor, there is no error; instead, the Parameter takes on the first value of the 1D tensor without complaint. This behavior is a bit surprising to me, as I expected a RuntimeError. See the following code for an example; the assertion passes:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
import torch.nn as nn


class SimpleModule(nn.Module):
    def __init__(self, threshold_value):
        super().__init__()
        self.threshold = nn.Parameter(torch.tensor(threshold_value))
    
    def forward(self, x):
        return x

large_tensor = torch.randn(32000)
state_dict = {"threshold": large_tensor}

module = SimpleModule(0.0)
module.load_state_dict(state_dict)
assert module.threshold.item() == state_dict['threshold'][0].item() 
</code></pre>
<p>Is this behavior intentional / documented?</p>

KFrank: <p>Hi Michael!</p>
<aside class="quote no-group" data-username="mh2parker" data-post="1" data-topic="223641" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/e19b73/48.png" class="avatar"> mh2parker:</div>
<blockquote>
<p>When I initialize a Parameter of a Module to be a scalar, and then load in a state_dict containing a 1D tensor, there is no error; instead, the Parameter takes on the first value of the 1D tensor without complaint.</p>
</blockquote>
</aside>
<p>I can reproduce your issue (on 2.9.0) using your example code.</p>
<p>This looks like a bug to me – you might consider filing a github issue.  (I don’t see this<br>
mentioned or documented anywhere.)</p>
<p>Note that <code>torch.tensor (0.0)</code> creates a pure-scalar “no-dimensions” tensor of <code>shape</code><br>
<code>torch.Size([])</code> (with value <code>0.0</code>).</p>
<p>As far as I can tell, this bug – of not flagging the shape mismatch – only occurs when trying<br>
to load a “regular” tensor into a <code>torch.Size([])</code> tensor.  Loading a <code>torch.Size([])</code> tensor<br>
into a “regular” tensor fails as expected with a size-mismatch error.</p>
<p>(In particular, loading a <code>torch.Size([])</code> scalar into a <code>torch.Size([1])</code> scalar fails.)</p>
<p>Best.</p>
<p>K. Frank</p>

mh2parker: <p>Thanks for taking a look and confirming that it works on 2.9.0! I’ve opened an issue <a href="https://github.com/pytorch/pytorch/issues/165873" rel="noopener nofollow ugc">here</a>.</p>
Get `state_dict` from `DataDistributedParallel` model while other thread is running `backward`

lsz: <p>Hi,</p>
<p>I am stuck with the following problem:</p>
<p>I have two ranks and each rank has two threads:</p>
<ol>
<li>(Learning Thread) Runs permanently <code>backward</code>on a <code>DataDistributedParallel</code> model</li>
<li>(Main Thread) Retrieves from time to time the <code>state_dict</code> of the model to broadcast it to other independent processes.</li>
</ol>
<p>The problem I encounter is that the processes stall. I have tried out many ways by setting <code>barrier</code>s, using thread locks, even using <code>torch.save</code> to a byte stream in memory and then loading it from it again. It appears to me that this setup is somehow not possible.</p>
<p>Did anyone ever encounter this? Is there a solution to it?</p>
<p>Btw: Running the second thread without getting the <code>state_dict</code> runs perfectly. So I guess that this indeed a problem induced by the DDP.</p>
[solved] Trouble opening pytorch doc just now

moonlightlane: <p>I was trying to access the doc just now but the webpage either appears blank or looks like plain html? Does anyone else have problem opening the docs? <a href="https://pytorch.org/docs/stable/index.html" rel="nofollow noopener">https://pytorch.org/docs/stable/index.html</a></p>

ngimel: <p>Yes, I have the same issue.</p>

moonlightlane: <p>I think it’s back to normal now!</p>

habteshbeki: <p>I’ve the same issue I’m also can’t access pytorch doc</p>
Brenier maps in Pytorch

ShimBam: <p>I’m trying to train a Brenier map using Pytorch.  A Brenier map is the gradient of a convex function, so I’m using Input Convex Neural Network (ICNN) to implement the convex function.  Here is my code.</p>
<pre><code class="lang-auto">class ICNN(nn.Module):
    def __init__(self, dim, layers):
        super(ICNN, self).__init__()
        self.dim = dim
        self.layers = layers
        self.Wbx = []
        for _ in range(0,layers-1):
            self.Wbx.append(nn.Linear(dim, dim, bias=True))
        self.Wbx.append(nn.Linear(dim, 1, bias=True))

        self.Wz = []
        for _ in range(1,layers-1):
            self.Wz.append(nn.Linear(dim, dim, bias=False))
        self.Wz.append(nn.Linear(dim, 1, bias=False))
        self.clip()

    def clip(self):
        for W in self.Wz:
            with torch.no_grad():
                W.weight[W.weight&lt;0.0] = 0.0

    def forward(self, x):
        z = torch.pow(nn.functional.leaky_relu(self.Wbx[0](x), 0.2), 2.0)
        for i in range(1,self.layers):
            z = nn.functional.leaky_relu(self.Wz[i-1](z) + self.Wbx[i](x), 0.2)
        return z


class BRENIER(nn.Module):
    def __init__(self, dim, layers):
        super(RENIER, self).__init__()
        self.dim = dim
        self.layers = layers
        self.icnn = ICNN(dim, layers)

    def clip(self):
        self.icnn.clip()
    
    def forward(self, x):
        z = self.icnn(x)
        dzdx = torch.autograd.grad(outputs=z, inputs=x, create_graph=True)[0]
        return dzdx
</code></pre>
<p>When processing a batch of data, I get the error in the “torch.autograd.grad” call in “forward”.</p>
<p>“RuntimeError: grad can be implicitly created only for scalar outputs”</p>
<p>I understand that autograd.grad wants to be called on a scalar function, so how should I set this up so that autodifferentiation will work on this gradient?</p>

ptrblck: <p>Either reduce the output tensor to a scalar or provide the gradient explicitly in the same shape as the output tensor.</p>
<p>Besides that your code shows other issues since you are not properly registering the modules by using plain <code>list</code>s:</p>
<pre><code class="lang-auto">class ICNN(nn.Module):
    def __init__(self, dim, layers):
        super(ICNN, self).__init__()
        self.dim = dim
        self.layers = layers
        self.Wbx = []
        for _ in range(0,layers-1):
            self.Wbx.append(nn.Linear(dim, dim, bias=True))
        self.Wbx.append(nn.Linear(dim, 1, bias=True))

        self.Wz = []
        for _ in range(1,layers-1):
            self.Wz.append(nn.Linear(dim, dim, bias=False))
        self.Wz.append(nn.Linear(dim, 1, bias=False))
</code></pre>
<p>Use <code>nn.ModuleList</code> instead to make sure the linear layers are registered.</p>
Nvidia rtx 5070 with CUDA 13

Leszek_Jurkowski: <p>Which versions should I install to get support for my 5070 rtx (12 gb vram) with CUDA v13? So far I am getting errors, even if I try to compile the sources myself…</p>
<p>Thanks in advance!</p>

ptrblck: <p>You can install any PyTorch binary built with CUDA 12.8+ for Blackwell support.</p>

_zcy: <p>I think, always use the latest version of CUDA is very dangerous, because it may cause some issue very very hard to solve, just become some small API changes… Solve these bug is really…</p>
<p>Maybe you can use the “nightly builds”, <code>pip3 install --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/cu130</code>, because MAYBE some of nightly builds perormances better in fitting new versions of CUDA than stable releases: <code>pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu130</code></p>
<p>But nightly builds will cause some bad results ABSOLUTELY, like they may calc slowly, or they need some special DLLs. I think, the best plan, is ALWAYS WAIT FOR THE (NEXT) STABLE BUILD THAT MAY FIT YOU.</p>

ptrblck: <p>I disagree with this statement as generally the latest CUDA toolkit will perform the best for the latest hardware generation. Especially for Blackwell GPUs I would try our latest stack.</p>
<p>If you are seeing any issues, please let us know on GitHub.</p>
CUDA profiler not working on Windows

MLML: <p>Hi, I am new to pytorch and trying to set up a pytorch + GPU env on windows but currently stuck in a state where CUDA works (torch.cuda.is_available() = true) but torch.profiler does not work due to the following error: “WARNING:2025-10-17 22:41:06 20088:17884 C:\actions-runner_work\pytorch\pytorch\pytorch\third_party\kineto\libkineto\src\init.cpp:107] function cbapi-&gt;getCuptiStatus() failed with error CUPTI_ERROR_INVALID_DEVICE (2) WARNING:2025-10-17 22:41:06 20088:17884 C:\actions-runner_work\pytorch\pytorch\pytorch\third_party\kineto\libkineto\src\init.cpp:108] CUPTI initialization failed - CUDA profiler activities will be missing INFO:2025-10-17 22:41:06 20088:17884 C:\actions-runner_work\pytorch\pytorch\pytorch\third_party\kineto\libkineto\src\init.cpp:110] If you see CUPTI_ERROR_INSUFFICIENT_PRIVILEGES, refer to <a href="https://developer.nvidia.com/nvidia-development-tools-solutions-err-nvgpuctrperm-cupti" title="https://developer.nvidia.com/nvidia-development-tools-solutions-err-nvgpuctrperm-cupti" rel="noopener nofollow ugc">https://developer.nvidia.com/nvidia-development-tools-solutions-err-nvgpuctrperm-cupti</a>”. My env: nvidia-smi output: | NVIDIA-SMI 573.24 Driver Version: 573.24 CUDA Version: 12.8 |. pip show torch result "pip show torch Name: torch Version: 2.9.0+cu128  In addition, the CUPTI files are also confirmed there and the path is put in the PATH. C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.0\extras\CUPTI\lib64. So i really do not know which part can go wrong. Can someone help take a look? Thanks</p>
Congratualtions to @ptrblck

vdw: <p>I’m not sure if this counts as site feedback, but I would like to congratulate <a class="mention" href="/u/ptrblck">@ptrblck</a> on winning the PyTorch 2023 Award for “Excellence in All Aspects of Community Contributions”. And I’ll be damned if this is not more than well deserved! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>Thanks for the almost unbelievable amount of time and effort spent helping guys like me in this forum! And I selfishly hope you will keep up the awesome work!</p>

J_Johnson: <p>Congratulations, <a class="mention" href="/u/ptrblck">@ptrblck</a> ! Well earned.</p>

ptrblck: <p>Thank you for the kind words! The forum wouldn’t be the same without all of you contributing so also a big thanks to you for being awesome!<br>
I obviously haven’t met every active member in real life, but it often still feels as if we’ve known each other for years already <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

YoungSheldon: <p>On god I see this dude more than my family</p>
Updating Selected parameters in each epoch

shivkrishnajaiswal: <p>Hi,</p>
<p>I’m using <strong>PyTorch 2.8.0</strong> to solve an optimisation problem with the <strong>Adam</strong> optimiser.<br>
In each epoch, I would want to select a subset of parameters to update, while keeping the rest unchanged.</p>
<p>Setting the gradients of unselected parameters to zero works for SGD-type optimisers, but not for Adam, since it also updates internal moment estimates (first and second moments) even when <code>grad = 0</code>.</p>
<p>The expected behaviour is that for unselected parameters in an epoch, both their moment estimates and corresponding parameter values remain unchanged.</p>
<p>Could anyone suggest a way to achieve this behaviour?</p>
<p>Below is the sample code I tried,</p>
<pre><code class="lang-auto">def setting_seed_in_torch(seed):
    import random

    import numpy as np
    import torch

    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)

    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False


setting_seed_in_torch(1)

x = torch.rand(1000, 5, dtype=torch.float64)
y = torch.rand(1000, 1, dtype=torch.float64)

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Linear(5, 1)

    def forward(self, x):
        return torch.relu(self.fc(x))


model = Net().to(dtype=torch.float64)

optimizer = optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.MSELoss()

def zero_out_gradient_elements(indices):
    w = model.fc.weight
    
    with torch.no_grad():
        for i in indices:
            w.grad[0, i] = 0.

for epoch in range(10):
    optimizer.zero_grad()
    indices = sorted(np.random.permutation(5)[:2])
    
    y_hat = model(x)
    loss = criterion(y_hat, y)
    loss.backward()
    
    zero_out_gradient_elements(indices)
    
    print(f'{indices=}')
    print('before -&gt; ', model.fc.weight.ravel().numpy(force=True))
    optimizer.step()
    print('after  -&gt; ', model.fc.weight.ravel().numpy(force=True))
    print('*' * 50)
    
</code></pre>

ptrblck: <p>The cleanest approach might be to restore the “frozen” parameters after the <code>optimizer.step()</code> updated them with the running stats.</p>
For loop F.cosine_similarity make OOM

XiXiRuPan: <p>Why does the for loop calculation of F.cosine_similarity cause OOM of GPUmemory? The first for loop uses 18000 GPU memory, but the 200th loop uses 60000 GPU memory. Why is this? Please help. Thank you.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0bb596bf9d3286328fbe53b5ab59a8ec8f514e6c.png" data-download-href="https://discuss.pytorch.org/uploads/default/0bb596bf9d3286328fbe53b5ab59a8ec8f514e6c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0bb596bf9d3286328fbe53b5ab59a8ec8f514e6c.png" alt="image" data-base62-sha1="1FAiSyGsjg43mpfaZxAsfWP4n6c" width="690" height="385" data-dominant-color="282826"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1107×618 44.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

albanD: <p>Hey!</p>
<p>That is a great question why the 200th loop would make a jump in memory.</p>
<p>cc <a class="mention" href="/u/fffrog">@FFFrog</a> for torch_npu maybe?</p>

FFFrog: <p><a class="mention" href="/u/alband">@albanD</a>  Thank you for mention.</p>
<p>Yes, this maybe bugs of torch_npu, I will figure it out.</p>

FFFrog: <p>Hey <a class="mention" href="/u/xixirupan">@XiXiRuPan</a> , could you mind file a issue in this <a href="https://gitcode.com/Ascend/pytorch" rel="noopener nofollow ugc">repo</a>, since this is not a PyTorch bug but a torch_npu bug, and I will find the root cause and show it in the question</p>
When will sm120 support be available?

FoxPe: <p>I have a 5070TI, and I’m actually tired of trying to build libraries from source code, downloading a new nightly version every evening, and testing it with many other frameworks. I was hoping that version 2.9.0 stable would add this support, but unfortunately, it hasn’t. I’m also still getting endless errors from all the other libraries at the same time. In this context, I would really like to know when full support for 50xx series graphics cards will be available.</p>

ptrblck: <p>PyTorch 2.7.0 already added Blackwell support on our PyTorch wheels built with CUDA 12.8 and nightly binaries were available even beforehand. There is no need to source build PyTorch and you can install any of our recent releases with CUDA 12.8+.</p>

FoxPe: <p>Yeah, you were right. I installed PyTorch Stable 2.9.0 (with CUDA 12.8), and the GPU was successfully detected</p>
<p>However, when I tried to test XTTS, a new problem arose. It requires torchcodec, but as far as I understand, its latest version, torchcodec 0.7, is not compatible with PyTorch 2.9.0</p>
<p>Do I need to install version 2.8.0?</p>

ptrblck: <p>The latest <code>torchcodec</code> release is <code>0.8</code> as seen <a href="https://pypi.org/project/torchcodec/#files">here</a> which is compatible with the latest <code>torch==2.9.0</code> release as described <a href="https://github.com/meta-pytorch/torchcodec?tab=readme-ov-file#installing-cpu-only-torchcodec">here</a>.</p>

FoxPe: <p>Now I see. It turns out that torchcodec was updated to 0.8 yesterday, but I saw the repository before the updated version 0.8. I will try the new version of torchcodec now.</p>
<p>Edited: Yes. It’s really working now. All I had to do was wait for the torchcodec repository to be updated. Thanks for your help!</p>

ptrblck: <aside class="quote no-group" data-username="FoxPe" data-post="5" data-topic="223621">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/f/f9ae1b/48.png" class="avatar"> FoxPe:</div>
<blockquote>
<p>Edited: Yes. It’s really working now. All I had to do was wait for the torchcodec repository to be updated. Thanks for your help!</p>
</blockquote>
</aside>
<p>That’s great to hear! Let us know in case you have any other questions or unexpected issues.</p>
Compiling a loop on MPS - exceeding limit of buffers

maciejkorzepa: <p>Hi,</p>
<p>I’m working on compiling a loop on MPS device, but I’m hitting an error “error: number of constant buffers exceeds maximum supported (31)” while compiling the backward pass. Forward pass compiles fine. I realize this is currently a limitation of Apple Metal, but given that compilation of the forward pass works fine, I feel it should be also feasible to compile the backward pass. I guess it’s due to how torch inductor works, not taking into account the buffer limit. Is there currently some way to fix this problem? (I tested this on cuda with up to 1000 iterations and it works fine there)</p>
<pre data-code-wrap="python"><code class="lang-python">import torch


class Model(torch.nn.Module):
    def __init__(self, n_iter):
        super().__init__()
        self.linear = torch.nn.Linear(64, 64)
        self.n_iter = n_iter

    @torch.compile(fullgraph=False)
    def forward(self, x):
        for i in range(self.n_iter):
            x = torch.sin(self.linear(x))

        return x


def main(device, n_iter):
    model = Model(n_iter=n_iter).to(device)

    x = torch.nn.Parameter(torch.randn(8, 64, device=device))
    target = torch.randn(8, 64, device=device)

    y = model(x)
    loss = (y - target).pow(2).mean()
    loss.backward()  # -&gt; buffer limit error upon compilation
    print('Success')


if __name__ == '__main__':
    main(device='mps', n_iter=100)
</code></pre>
Average Hausdorff loss implementation?

che85: <p>Hi there,</p>
<p>does anyone of you know an open source implementation of the Average Hausdorff distance? I would like to try using it for some 3D volumetric data but don’t quite know how to get started with writing a loss function in pytorch.</p>
<p>I am helpful for any kind of input.</p>
<p>Thanks a lot.<br>
Christian</p>

yyifan: <p>Hi Christian,</p>
<p>The open source code of a CVPR 2019 paper (Locating Objects Without Bounding Boxes) may be helpful. Here is the <a href="https://github.com/javiribera/locating-objects-without-bboxes/blob/master/object-locator/losses.py" rel="nofollow noopener">url</a>.</p>
<p>Not sure whether this is still helpful you right now.</p>

che85: <p>Thanks for this reference. It seem that it’s not applied to 3D. Am I correct?</p>

Priyank_Pathak: <p>5 years late, but here is a more generic implementation</p>
<pre><code class="lang-auto">X1.shape == (B, N, D)
X2.shape == (M, D)

X2 = X2.unsqueeze(0).expand(B, M, D)   # (B, M, D)
dist = torch.cdist(X1, X2_exp) # B, N, M 

dh_x_y = dist.min(1)[0].mean(-1) # B, M --&gt; B 
dh_y_x = dist.min(2)[0].mean(-1) # B, N --&gt; B 
dist = dh_x_y + dh_y_x
</code></pre>