akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pytorch İnference

Mahmut_ERDEM: <p>Hello! I have tried everything to download the weight and checkpoint .pt files that are created while training my own AI model on RunPod. When I download them to my Linux system, they are automatically zipped, and a data folder and a data.pkl file are created inside the archive. This is useless for inferencing the model. I tried downloading it with a command and even zipped it before downloading, but none of these methods worked. This problem is very simple to solve, but I don’t know how. Can you please guide me?</p>

ptrblck: <aside class="quote no-group" data-username="Mahmut_ERDEM" data-post="1" data-topic="222291">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mahmut_erdem/48/77945_2.png" class="avatar"> Mahmut_ERDEM:</div>
<blockquote>
<p>This is useless for inferencing the model.</p>
</blockquote>
</aside>
<p>Could you describe your issue in more detail and what exactly does not work or any errors you are seeing in your workflow?</p>

Mahmut_ERDEM: <p>First, I uploaded the working files of my language model to the server. Then I trained it with H200Sxm. The weight.pt and checkpoint.pt files were created at the end of each epoch. I wanted to take these files and run them locally. Later, I also downloaded them using Colab commands. My .pt files were downloaded, but even though I didn’t compress them into a zip, they were downloaded as archives. Then I tried a different method, assuming this might be due to Chrome browser behavior — the file downloads, but my computer automatically treats .pt files as archive files by default. I also paid attention to this in my source code. Normally, shouldn’t a .pt file be a single file?</p>
Numerical mismatch between flash_attn_qkvpacked_func and flex_attention with token-level mask

davidnewman02: <p>I’ve been trying to substitute flash_attention for flex_attention in my project, but I find that the attention output is numerically different between the two, despite using what I believe is an equivalent masking strategy. This is sufficient to cause instabilities in training where losses become <code>NaN</code> during training when using flex_attention</p>
<p>My attempt at an equivalent implementation:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch

from torch.nn.attention.flex_attention import flex_attention, create_block_mask
from flash_attn import flash_attn_qkvpacked_func

WINDOW_SIZE = (127, 128)

def mask_mod(b, h, q_idx, kv_idx):
    return (kv_idx &gt;= (q_idx - WINDOW_SIZE[0])) &amp; (kv_idx &lt;= (q_idx + WINDOW_SIZE[1]))

def flex_attn_impl(qkv):
    N, T, _, H, D = qkv.shape
    q, k, v = qkv.permute(0, 2, 3, 1, 4).unbind(dim=1)
    block_mask = create_block_mask(mask_mod, N, H, T, T, device=q.device)
    attn_out = flex_attention(q, k, v, block_mask=block_mask)
    return attn_out.permute(0, 2, 1, 3)

def flash_attn_impl(qkv):
    return flash_attn_qkvpacked_func(qkv, window_size=WINDOW_SIZE)

def load_qkv():
    return torch.load("qkv.pt").to(torch.float16)

qkv = load_qkv()
flash_out = flash_attn_impl(qkv)
flex_out = flex_attn_impl(qkv)
print("QKV:  ", qkv.shape, qkv.sum())
print("Flash:", flash_out.shape, flash_out.sum())
print("Flex: ", flex_out.shape, flex_out.sum())

cmp_elements = flex_out == flash_out
print("equal:   ", torch.equal(flex_out, flash_out))
print("pct-eq:  ", cmp_elements.sum() / cmp_elements.numel())
print("max-diff: ", (flex_out - flash_out).abs().max())
</code></pre>
<p>My output from these two implementations is similar, but not identical:</p>
<pre data-code-wrap="python"><code class="lang-python">QKV:   torch.Size([64, 417, 3, 8, 64]) tensor(-10976., device='cuda:0', dtype=torch.float16, grad_fn=&lt;SumBackward0&gt;)
Flash: torch.Size([64, 417, 8, 64]) tensor(6248., device='cuda:0', dtype=torch.float16, grad_fn=&lt;SumBackward0&gt;)
Flex:  torch.Size([64, 417, 8, 64]) tensor(6248., device='cuda:0', dtype=torch.float16, grad_fn=&lt;SumBackward0&gt;)

equal:     False
pct-eq:    tensor(0.7966, device='cuda:0')
max-diff:  tensor(0.0010, device='cuda:0', dtype=torch.float16, grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
<p>I’ve tried setting <code>block_mask(…, BLOCK_SIZE=1)</code> to avoid any block-level mask coarsening, but this gives me identical results.</p>
<p>Is this numerical difference expected due to implementation differences or is there something I’m overlooking in how <code>flex_attention</code> applies the mask that I could try?</p>

davidnewman02: <p>Versions:</p>
<ul>
<li>python==3.12.3</li>
<li>torch==2.7.1+cu128</li>
<li>flash_attn==2.8.0.post2 (flash_attn-2.8.0.post2+cu12torch2.7cxx11abiFALSE-cp312-cp312-linux_x86_64.whl)</li>
</ul>
Request to Share the Survey with your Software Developers and Code Reviewers

Niaz: <p>Dear Sir,</p>
<p>I hope this message finds you well.</p>
<p>My name is Md Niaz Morshed , and I am leading a team conducting a research study titled <strong>“Understanding Information Needs When Looking for Security Issues During Code Review in Open-Source Software (OSS): A Survey."</strong></p>
<p>The purpose of this study is to better understand how developers identify and address security issues during peer code review in open-source projects. We aim to explore the types of information developers rely on, the challenges they face, and where support mechanisms can be improved. We invite you to participate in an anonymous survey that will take approximately 20-25 minutes to complete. Your responses will contribute to a broader understanding of current practices and help identify opportunities to improve security-focused code review.</p>
<p>Participation is entirely voluntary, and you may withdraw at any time. No personally identifiable information will be collected. Upon completing the survey, you will have the option to enter a raffle for a chance to receive one of two $50 gift cards. Participation in the raffle is voluntary and will require you to visit a separate survey to provide your email address. In this way, your email address will not be associated with survey responses. Winners will be selected at random and notified via email after the survey concludes. The research has been approved by the University of Alabama Institutional Review Board.</p>
<p>I would greatly appreciate it if you could share the survey link below with your project’s developers and code reviewers. Thank you in advance.</p>
<p><a href="https://universityofalabama.az1.qualtrics.com/jfe/form/SV_03zTHf83SiO01f0" title="https://universityofalabama.az1.qualtrics.com/jfe/form/SV_03zTHf83SiO01f0" rel="noopener nofollow ugc">https://universityofalabama.az1.qualtrics.com/jfe/form/SV_03zTHf83SiO01f0</a></p>
<p>Sincerely yours</p>
<p>Niaz</p>
<p><strong>Md Niaz Morshed</strong></p>
<p><em>Graduate Research Assistant,</em></p>
<p><em>Department of Computer Science,</em></p>
<p><em>University of Alabama, Tuscaloosa,</em></p>
<p><em>USA.</em></p>
RunTimeError: Trying to create tensor with negative dimension at loss.backward()

AeroDEmi: <p>I have a VAE that I want to train but at <code>loss.backward()</code>,  it shows:<br>
<code>RuntimeError: Trying to create tensor with negative dimension -17146298352: [-17146298352]</code></p>
<p>Here is part of my code:</p>
<pre><code class="lang-auto">for epoch in range(NUM_EPOCHS):
    torch.cuda.empty_cache()
    loop = tqdm(enumerate(train_loader))
    for i, (image, x_tensor, y_tensor) in loop:
        x_latent = vae.encode(x_tensor.to('cuda').half())
        x_latent = x_latent.latent_dist.sample() * vae.config.scaling_factor

        y_reconstructed = vae.decode(x_latent, return_dict=False)[0]
        loss = loss_fn(y_reconstructed, y_tensor.to('cuda').half())
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        loop.set_postfix(loss=loss.item())
</code></pre>
<p>Both <code>y_reconstructed</code> and <code>y_tensor</code> have the same shape <code>[1, 4, 1024, 1024]</code></p>
<p>This is the output of the loss variable:</p>
<pre><code class="lang-auto">tensor(0.4946, device='cuda:0', dtype=torch.float16,
       grad_fn=&lt;MseLossBackward0&gt;)
</code></pre>
<p>My model is from HF AutoEncoderKL:</p>
<pre><code class="lang-auto">vae = AutoencoderKL.from_pretrained('madebyollin/sdxl-vae-fp16-fix', torch_dtype=torch.float16, use_safetensors=True)
vae.decoder.conv_out = torch.nn.Conv2d(128, 4, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
vae.encoder.requires_grad_(False)
vae.to("cuda")
</code></pre>

chenyx.cs: <p>i meet the same question! have you solve it?   I am finetuning a vae too</p>

breengles: <p>Disabling mixed precision helped me</p>

chenyx.cs: <p>you mean disabling accelerate’s mixed precision? and load sdxl-vae-fp16-fix in fp16?</p>

breengles: <p>disabling accelerate’s mp and loading in fp32 but this is for vae from sd1.5 so dunno if it is the case for sdxl</p>

Jim_Yan: <p>same error， use accelerator give the error</p>
RESENET-50 -- Custom Architecture

Smit_Desai: <p>Hi, from the past few months, i am working on resnet-50 which almost works good , but have some confusion regarding my model training and my resnet architecture. Can you guys help me. Like is this proper architecture or needs some tweaking??</p>
<p>class SelfAttention(nn.Module):<br>
def <strong>init</strong>(self, in_dim, heads=4, dropout=0.1):<br>
super(SelfAttention, self).<strong>init</strong>()<br>
self.heads = heads<br>
self.query = nn.Linear(in_dim, in_dim, bias=False)<br>
self.key = nn.Linear(in_dim, in_dim, bias=False)<br>
self.value = nn.Linear(in_dim, in_dim, bias=False)<br>
self.dropout = nn.Dropout(dropout)<br>
self.softmax = nn.Softmax(dim=-1)<br>
self.norm = nn.LayerNorm(in_dim)</p>
<pre><code class="lang-auto">def forward(self, x):
    B, C, H, W = x.shape
    x_orig = x
    x = x.view(B, C, -1).permute(0, 2, 1)  # (B, N, C)

    Q = self.query(x)
    K = self.key(x)
    V = self.value(x)

    attn_scores = torch.bmm(Q, K.transpose(1, 2)) / (C ** 0.5)
    attn_scores = self.softmax(attn_scores)
    attn_scores = self.dropout(attn_scores)

    attn_output = torch.bmm(attn_scores, V)
    attn_output = self.norm(attn_output + x)  # Residual + Norm

    return attn_output.permute(0, 2, 1).view(B, C, H, W) + x_orig
</code></pre>
<h1><a name="p-474101-h-1" class="anchor" href="#p-474101-h-1"></a>-------------------------</h1>
<h1><a name="p-474101-custom-resnet-with-attention-2" class="anchor" href="#p-474101-custom-resnet-with-attention-2"></a>Custom ResNet with Attention</h1>
<h1><a name="p-474101-h-3" class="anchor" href="#p-474101-h-3"></a>-------------------------</h1>
<p>class CustomResNet(nn.Module):<br>
def <strong>init</strong>(self, num_classes=100, dropout=0.3, freezing_layers=True, resnet_model=‘resnet50’):<br>
super(CustomResNet, self).<strong>init</strong>()<br>
backbone = timm.create_model(resnet_model, pretrained=True)</p>
<pre><code class="lang-auto">    # Freeze early layers if needed
    if freezing_layers:
        for name, param in backbone.named_parameters():
            if "layer4" not in name:
                param.requires_grad = False

    self.backbone = nn.Sequential(*list(backbone.children())[:-2])
    self.attention = SelfAttention(in_dim=2048)

    self.conv_layers = nn.Sequential(
        nn.Conv2d(2048, 1024, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(1024),
        nn.ReLU(),
        nn.Conv2d(1024, 512, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(512),
        nn.ReLU(),
        nn.AdaptiveAvgPool2d((1, 1))
    )

    self.fc_layers = nn.Sequential(
        nn.Flatten(),
        nn.Linear(512, 256),
        nn.BatchNorm1d(256),
        nn.ReLU(),
        nn.Dropout(dropout),
        nn.Linear(256, num_classes)
    )

def forward(self, x):
    resnet_output = self.backbone(x)  # (B, 2048, 7, 7)
    attn_result = self.attention(resnet_output)
    x = self.conv_layers(attn_result)
    x = self.fc_layers(x)
    return x, resnet_output, attn_result
</code></pre>
Defect in Libtorch c++ installation

S_Hamzeh74: <p>Hi<br>
I have downloaded the libtorch 2.7.1-cu118 version from <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">this</a> link.</p>
<p>I have added the include directory on the Visual Studio 2022 in the C/C++ → General → Additional Include Directories:</p>
<ul>
<li>C:\Libtorch\include</li>
<li>C:\Libtorch\include\torch\csrc\api\include</li>
</ul>
<p>and in the Linker → General → Additional Library Directories:</p>
<ul>
<li>C:\Libtorch\lib</li>
</ul>
<p>and in the Input → Additional Dependencies:</p>
<ul>
<li>asmjit.lib<br>
c10.lib<br>
c10d.lib<br>
c10_cuda.lib<br>
caffe2_detectron_ops_gpu.lib<br>
caffe2_module_test_dynamic.lib<br>
caffe2_nvrtc.lib<br>
clog.lib<br>
cpuinfo.lib<br>
dnnl.lib<br>
fbgemm.lib<br>
gloo.lib<br>
gloo_cuda.lib<br>
libprotobuf-lite.lib<br>
libprotobuf.lib<br>
libprotoc.lib<br>
mkldnn.lib<br>
torch.lib<br>
torch_cpu.lib<br>
torch_cuda.lib</li>
</ul>
<p>also add the C:\Libtorch\lib directory to the system Environment Variables.</p>
<p>this is my sample code:</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">#include&lt;opencv2/opencv.hpp&gt;
#include &lt;torch/torch.h&gt;
#include &lt;torch/script.h&gt;
</code></pre>
<pre data-code-wrap="cpp"><code class="lang-cpp">int main(){
//cuda
auto device = torch::Device(torch::kCUDA,0);//read pic
auto image = cv::imread(“your path to\flower.jpg”);//resize
cv::resize(image, image, cv::Size(224, 224));//convert to tensor
auto input_tensor = torch::from_blob(image.data, { image.rows, image.cols, 3 }, torch::kByte).permute({ 2, 0, 1 }).unsqueeze(0).to(torch::kFloat32) / 225.0;//load model
auto model = torch::jit::load(“your path to\resnet34.pt”);
model.to(device);model.eval();//forward the tensor
auto output = model.forward({input_tensor.to(device)}).toTensor();
output = torch::softmax(output, 1);
std::cout &lt;&lt; "predicted class: " &lt;&lt; torch::argmax(output) &lt;&lt; ", prob is: " &lt;&lt; output.max() &lt;&lt; 
std::endl;return 0;
}
</code></pre>
<p>but only options I found in the visual studio is these:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b53dd607061802a4e3c622802bbbdd3552e02bf4.png" data-download-href="https://discuss.pytorch.org/uploads/default/b53dd607061802a4e3c622802bbbdd3552e02bf4" title="torch"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b53dd607061802a4e3c622802bbbdd3552e02bf4.png" alt="torch" data-base62-sha1="pRkU7QepSA38y2GM9xEryy4cO2g" width="380" height="347"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">torch</span><span class="informations">380×347 12.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I cannot find torch::from_blob, torch::Device , torch::jit::load and any other things…<br>
what can I do ?<br>
How can I install in use libtorch c++ properly ?</p>
How to convert MXFP4 -> FP8 in pure pytorch?

vgoklani: <p>We have four NVIDIA A6000 Ada cards which are all SM89 and don’t support MXFP4. We would like to run GPT-OSS 20B (or any model actually) in FP8 using tensor parallel for inference. How should we convert the FP4 tensors to FP8 using just pytorch?</p>
<p>Thanks!</p>
Building/including PyTorch in a monorepo with Bazel

jteuwen: <p>We are working in a large Bazel-driven monorepo where we both have C++ bits using libtorch’s dynamic libraries and python code where we’re using PyTorch (using pip). However, when we try to bind the C++ parts using pybind11 to python, we do get a conflict in library resolution and symbols not being found. After all the dynamic libraries provided by libtorch and PyTorch appear to be slightly different.</p>
<p>Clearly a single source of truth is needed. PyTorch seems to come with a BUILD.bazel file, but it doesn’t appear you can build PyTorch using it. Is this correct? Did anyone succeed building PyTorch using Bazel? Any ideas how to address making sure both my C++ libtorch and PyTorch versions are linked to the same shared objects?</p>
Getting error in installing pytorch with Open MPI

Shubham_Kakde: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/e/aed2b4c3e1e21511ea06173322bf69f60e418375.png" data-download-href="https://discuss.pytorch.org/uploads/default/aed2b4c3e1e21511ea06173322bf69f60e418375" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/aed2b4c3e1e21511ea06173322bf69f60e418375_2_690x76.png" alt="image" data-base62-sha1="oWyvY3VA8XrPwcBCHO1RADRmV1j" width="690" height="76" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/e/aed2b4c3e1e21511ea06173322bf69f60e418375_2_690x76.png, https://discuss.pytorch.org/uploads/default/original/3X/a/e/aed2b4c3e1e21511ea06173322bf69f60e418375.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/a/e/aed2b4c3e1e21511ea06173322bf69f60e418375.png 2x" data-dominant-color="262626"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1034×114 36.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div>I’m encountering an error while installing PyTorch (v2.1) with OpenMPI(5.0.8). Any help would be greatly appreciated</p>

ptrblck: <p>The failing line of code can neither be found in the current <a href="https://github.com/pytorch/pytorch/blame/7ae0629d64b404e0ef5d9c931433ad25e65d6114/torch/csrc/distributed/c10d/ProcessGroupMPI.cpp#L39-L47"><code>main</code> branch</a> nor in <a href="https://github.com/pytorch/pytorch/blame/v2.1.0/torch/csrc/distributed/c10d/ProcessGroupMPI.cpp#L38-L47">2.1.0</a>. Did you add this code yourself, did you cherry-pick a PR from another branch, or could you let me know where this code comes from?</p>
Request for SM_120 / RTX 50xx (CUDA 12.8) Support in PyTorch

addictedsociety: <p>Hi everyone,</p>
<p>I just got a new PC with an RTX 5090 (SM 12.0, CUDA 12.8) and wanted to set up ComfyUI with GPU acceleration using PyTorch.<br>
While following the usual installation process, I noticed that the current stable PyTorch builds don’t seem to include support for SM 12.0 yet.</p>
<p>From what I’ve read so far, the options seem to be:</p>
<ul>
<li>
<p>Waiting for official builds that support SM 12.0</p>
</li>
<li>
<p>Compiling PyTorch from source with <code>TORCH_CUDA_ARCH_LIST="12.0"</code></p>
</li>
<li>
<p>Using unofficial wheels shared by community members</p>
</li>
</ul>
<p>Before I go down the manual build route, I wanted to ask:</p>
<ul>
<li>
<p>Is there an official timeline for SM 12.0 support in PyTorch?</p>
</li>
<li>
<p>Are there any nightly builds or test versions I can try now?</p>
</li>
<li>
<p>What’s currently the recommended approach for running PyTorch on RTX 50xx GPUs?</p>
</li>
</ul>
<p>Thanks in advance!</p>

ptrblck: <p>All of our stable (2.7 and 2.8) and nightly releases built with CUDA 12.8+ support Blackwell architectures.</p>
Is there a way to install cpp_extension only?

victbr: <p>I am attempting to package my project using <code>pyproject.toml</code> with <code>setuptools</code> as the backend. For building C/C++ extensions, I utilize <code>BuildExtension</code> and <code>CppExtension</code> from <code>torch.utils.cpp_extension</code> within <code>setup.py</code>. Consequently, I must include <code>torch</code> in the <code>requires</code> section of the <code>[build-system]</code> in <code>pyproject.toml</code>. I understand that the build frontend, such as <code>pip</code>, creates a new environment to install the dependencies listed in <code>setup.py</code>, which necessitates the installation of <code>torch</code> in this temporary environment. As far as I known, this process is designed to maintain a clean final environment, potentially leading to repeated installations.</p>
<p>The issue I encounter is the prolonged duration of the <code>installing build dependencies</code> phase, despite <code>torch</code> already being present in the current environment. I suspect this inefficiency stems from the aforementioned process of repeatedly installing <code>torch</code> in a new, isolated environment.</p>

ice-tong: <p>I have the same question!</p>
Caffe2 support (python)

sbuslovsky: <p>I have a (quite old) application that was built with Caffe2 prior to it’s merger into Pytorch.</p>
<p>As per the guide on caffe2.ai, it seems to suggest that caffe2 libraries have been integrated to Pytorch.</p>
<p>I installed pytorch through conda.<br>
But still I can’t import the libraries to import caffe2 prebuilt modes (in protobuf format).</p>
<p>Specific error:</p>
<pre data-code-wrap="python"><code class="lang-python">from caffe2.python import core, workspace, visualize, model_helper
</code></pre>
<p><strong>ModuleNotFoundError: No module named ‘caffe2’</strong></p>
<p>(code is based of <a href="https://caffe2.ai/docs/tutorial-loading-pre-trained-models.html" rel="noopener nofollow ugc">Loading Pre-Trained Models</a> guide)</p>
<p>Q: What is the right way to run caffe2 applications?</p>

sbuslovsky: <p>I found the PR where this was done: <a href="https://github.com/pytorch/pytorch/pull/126035" class="inline-onebox" rel="noopener nofollow ugc">Remove Caffe2 python code by cyyever · Pull Request #126035 · pytorch/pytorch · GitHub</a>.</p>
<p>Seems like a breaking change that invalidates most of the documentation on caffe2.ai that’s still out there.</p>
Portable Whisper's torch Folder 7GB > Win11 ISO

bytos: <p>Is there a way to reduce the size of the torch folder (6.91GB)?</p>
<p>The Whisper model used is only 461MB.</p>
<p>Whisper is used to create subtitles from audio files with a Blackwell GPU.</p>
<p>PyTorch 2.8<br>
CUDA 12.8</p>
Problems in compiling and installing libtorch source code

strongspirit-fighter: <p>Now I have the arm board of Jetson. Now I have downloaded the official source code libtorch file. Now I want to compile and install this library. This figure is my file structure. Let me first create a build folder in the folder and execute cmake .. In the build folder; Then execute make install; Is it the correct way for me to install the libtorch GPU library correctly?Thank you very much!!!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/5/d57c21d4bfdd9f648126f279468a347062d8ee7d.png" data-download-href="https://discuss.pytorch.org/uploads/default/d57c21d4bfdd9f648126f279468a347062d8ee7d" title="图片"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d57c21d4bfdd9f648126f279468a347062d8ee7d_2_690x48.png" alt="图片" data-base62-sha1="uszEuzOoXpJrJYTUdRqqwXpoNGt" width="690" height="48" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d57c21d4bfdd9f648126f279468a347062d8ee7d_2_690x48.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d57c21d4bfdd9f648126f279468a347062d8ee7d_2_1035x72.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/5/d57c21d4bfdd9f648126f279468a347062d8ee7d_2_1380x96.png 2x" data-dominant-color="391730"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">图片</span><span class="informations">1798×126 38.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
`torch.compile` (w/ Torch Inductor) benchmarks/models for Multi GPU

Abhishek_Ghosh: <p>I have understanding of the models from the <a href="https://docs.pytorch.org/assets/pytorch2-2.pdf" rel="noopener nofollow ugc">PyTorch 2.0 paper</a> using <code>torch.compile</code>, which use the default Inductor backend. The PyTorch <a href="https://github.com/pytorch/pytorch/tree/main/benchmarks/dynamo" rel="noopener nofollow ugc">benchmark/CI</a> has scripts to test them out. But they all seem to make use of <code>torch.compile</code> for a single GPU. Any benchmark/model script to get started with using <code>torch.compile</code> for Multi-GPU? And how to get started along that direction?</p>
<p>I know <a href="https://docs.vllm.ai/en/latest/design/torch_compile.html" rel="noopener nofollow ugc">vLLM v2</a> makes use of <code>torch.compile</code>, but it seems to be somewhat a different (custom) backend than Torch Inductor. (It makes use of portions of Torch Inductor – a wrapper around the Torch Inductor element for its backend).</p>
<p>I wanted to explore few Multi-GPU models deployed using Torch Inductor of <code>torch.compile</code>.</p>
Is NVCleanstall safe to use? Any better options?

jhonsip: <p>Hey everyone,</p>
<p>I want to install just the basic Nvidia drivers without all the extra stuff. I found a tool called <a href="https://nvcleanstal.com/" rel="noopener nofollow ugc">NVIDIA clean install</a>—is it safe and reliable?</p>
<p>Also, are there any better or easier tools out there?</p>
<p>Main goal is keeping things clean and making sure <strong>OBS with NVENC still works</strong>.</p>
<p>Thanks!</p>

ptrblck: <p>I’m not familiar with this tool and don’t know who built it.</p>
<p>If you want to install the NVIDIA drivers only, I’m wondering why the <a href="https://www.nvidia.com/en-us/drivers/">NVIDIA Driver install page</a> would not work?</p>
Control over 'original' tensor in parametrizations

jmitch: <p>It seems like torch’s nn.utils.parametrize module and ‘register_parametrization’ does not offer the ability to directly set the shape of the resulting parameter tensor. There is very limited control if you are able to define a right_inverse method in your parametrization class - if you return multiple tensors from right_inverse, then those tensors are set as the underlying parameters (in weight.parametrization.original). If you don’t define a right inverse, it seems that register_parametrization assumes the right inverse is just the identity, in which case, the ‘original’ tensor (i.e. parameters that parametrize the new weights) will have the same shape as the weights being parametrized.</p>
<p>I’m not sure how to work around - in my use-case there isn’t a natural right inverse but I still need the ability to set the original parameters or at least define their shape. Maybe it’s possible to set it up in a really clunky way - e.g. if I have a weight matrix M that with shape [100,100] that needs to be parametrized by a set of 10 parameters (e.g. if you set it as a block-diagonal matrix with a specified weight for all elements of each block), you could set the first ten elements of M to the intended initial value of the underlying parameters, then define right_inverse so it returns M[0,:10]. And then in forward, I assume the ‘X’ arg will have shape [10,], and then you could build the parametrized M from these 10 parameters.</p>
<p>Does anyone know of a more elegant way to do this?</p>
<p>(for reference, I’m trying to reproduce the original Interactive Activation model, which is a dynamical/recurrent model where there are a small set of parameters defining the strengths of e.g. lateral inhibition, self-inhibition, between-layer excitation, etc.; all nodes at a given “level”, i.e. a block on the diagonal of the full adjacency matrix, are connected with the same weight - the weight specifying the strength of lateral inhibition in that level. I want to train this small set of parameters, but still use the original IA connectivity scheme.)</p>
VGGFace 2 - ResNet-50-128D pretrained model for Pytroch

Ruby_Shrestha: <p>The link for the pretrained models provided here <a href="https://github.com/ox-vgg/vgg_face2?tab=readme-ov-file" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ox-vgg/vgg_face2</a> do not work and I am looking for ResNet-50-128D pretrained model for Pytroch. From here <a href="https://discuss.pytorch.org/t/pytorch-1-0-loading-vggface2-weights-in-python3-7/40696" class="inline-onebox">PyTorch 1.0 loading VGGFace2 weights in Python3.7</a> I see that some have been able to download the weights but then this is from 2019.</p>
<p>It’d be great if someone could help me with the latest link for ResNet-50-128D. Also, given that the link from official github for it is not accessible now, was wondering whether we’re allowed to use it</p>
Pytorch support for sm_120

VIJAY_SANTHOSH_P: <p>i am using rtx 5090 and cuda 12.9 version but the pytorch 2.7 is not supporting the sm_120<br>
what should i do now<br>
NVIDIA GeForce RTX 5090 with CUDA capability sm_120 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_70 sm_75 sm_80 sm_86 sm_90.</p>

ptrblck: <p>Based on the error message you have installed an older binary or a PyTorch binary built with CUDA &lt;=12.6.</p>
<p>Install any stable or nightly PyTorch binary built with CUDA 12.8+ from our <a href="https://pytorch.org/get-started/locally/">install matrix</a> to support your Blackwell GPU.</p>
CUDA is ununavailable when using pytorch cpp version

strongspirit-fighter: <p>Hello, I have encountered a problem. I downloaded the source code of the cpp version of pytorch on my arm architecture computer. After compiling and installing, I tested the library. The result shows that cuda is not available, but it can be called by the CPU. At first, I thought it was because I didn’t mount the related files of cuda when I created the docker container, but this reason was ruled out by me, not this reason. NVCC – version is OK, and environment variables are OK.<br>
But I noticed a detail. This detail is that there is an ENABLE_CUDA: BOOL=OFF in CMakeCache.txt in the build folder of my pytorch, which is automatically generated after compilation. Now, can you help me analyze the reason why cuda is unavailable? If this is the reason, can you tell me how to solve it? Thank you!</p>
Libtorch nightly versions

juliusgh: <p>Hello,<br>
on <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">https://pytorch.org/get-started/locally/</a> it says that one should use the download link <a href="https://download.pytorch.org/libtorch/nightly/cu128/libtorch-cxx11-abi-shared-with-deps-latest.zip" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/nightly/cu128/libtorch-cxx11-abi-shared-with-deps-latest.zip</a> for the latest nightly release of libtorch.<br>
However, the last CXX11 ABI version is from March 20. All newer versions for Linux are published under a file name like libtorch-shared-with-deps-2.8.0.dev20250507%2Bcu128.zip.<br>
Does someone know if the CXX11 ABI variant is discontinued and why it is still recommended on  the PyTorch website?</p>
<p>Thanks in advance!</p>

ptrblck: <p>We stopped building binaries with pre-CXX11 ABI so you should be able to download the latest one without the <code>cxx11-abi</code> tag.<br>
We should revisit the install command and fix it. CC <a class="mention" href="/u/malfet">@malfet</a> for visibility</p>

juliusgh: <p>Perfect, thanks for the quick reply! The latest version without the <code>cxx-abi</code> tag also works flawlessly for me and has the latest commit from GitHub.</p>

juliusgh: <p>I just wanted to follow up on that. Currently, the URL suggested at <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">https://pytorch.org/get-started/locally/</a> for installing the latest nightly version of libtorch not only leads to an old version, but is also broken:</p>
<pre><code class="lang-auto">https://download.pytorch.org/libtorch/nightly/cu129/libtorch-cxx11-abi-shared-with-deps-latest.zip
</code></pre>
<p>It should be changed to the version without “cxx11-abi”:</p>
<pre><code class="lang-auto">https://download.pytorch.org/libtorch/nightly/cu129/libtorch-shared-with-deps-latest.zip
</code></pre>
<p>CC <a class="mention" href="/u/malfet">@malfet</a></p>

malfet: <p>Thank you for reporting, let me create an issue and we should fix the links before 2.8.0 release, see <a href="https://github.com/pytorch/pytorch/issues/159880" class="inline-onebox" rel="noopener nofollow ugc">Nightly libtorch links on website are incorrect · Issue #159880 · pytorch/pytorch · GitHub</a></p>

atalman: <p>HI <a class="mention" href="/u/juliusgh">@juliusgh</a> this is fixed <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">https://pytorch.org/get-started/locally/</a> Please validate. I do see correct link in nightly</p>
<pre><code class="lang-auto">https://download.pytorch.org/libtorch/nightly/cu129/libtorch-shared-with-deps-latest.zip
</code></pre>

juliusgh: <p>Hi <a class="mention" href="/u/atalman">@atalman</a>. Thanks, now I also see the correct link!</p>
FSDP.HYBRID_SHARD leads to parameter inconsistency between two DP replicas

acplus: <p>Hi, I have a question about FSDP.</p>
<p>When we set the device_mesh as <code>init_device_mesh('cuda', mesh_shape=(2, 1), mesh_dim_names=['ddp', 'fsdp'])</code>.  In other words, we use the FSDP.HYBRID_SHARD to train a model.</p>
<p>After I perform optimizer.step, the model weights between two DP replicas is not the same.</p>
<p>May I ask give some advices to fix it?</p>
Purpose and communication of set reshard_after_forward=int in fsdp2

shadowshadow: <p>Hello everyone, when using PyTorch FSDP, I have several questions about setting <strong><code>reshard_after_forward</code></strong> to an integer (e.g. <strong><code>8</code></strong>):</p>
<ul>
<li>
<p>What is purpose of specifying a shard size as an integer compared to the Boolean modes (True/False)? Is this practice reasonable, and why not simply start training with that shard size from the beginning?</p>
</li>
<li>
<p>Communication flow for backward &amp; the next forward pass, assuming initial sharding size is 32, and set <code>reshard_after_forward=8</code>  :</p>
<ul>
<li>
<p>After the forward pass, each rank will have 1/8 params of model</p>
</li>
<li>
<p>During backward, how are all-gather and reduce_scatter executed within those 8-GPU groups? Or only all-gather is done within 8 GPUs, reduce_scatter and all_reduce are done within 32 GPUs</p>
</li>
<li>
<p>For the next forward pass, does the original 32-shard layout is required again?</p>
</li>
</ul>
</li>
</ul>
PyTorch `torch.no_grad` vs `torch.inference_mode`

timgianitsos: <p>PyTorch has new functionality <code>torch.inference_mode</code> as of v1.9 which is “<a href="https://pytorch.org/docs/stable/generated/torch.inference_mode.html" rel="noopener nofollow ugc">analogous to</a> <code>torch.no_grad</code>… Code run under this mode gets better performance by disabling view tracking and version counter bumps.”</p>
<p>If I am just evaluating my model at test time (i.e. not training), is there any situation where <code>torch.no_grad</code> is preferable to <code>torch.inference_mode</code>? I plan to replace every instance of the former with the latter, and I expect to use runtime errors as a guardrail (i.e. I trust that any issue would reveal itself as a runtime error, and if it doesn’t surface as a runtime error then I assume it is indeed preferable to use <code>torch.inference_mode</code>).</p>
<p>More details on why inference mode was developed are mentioned in <a href="https://podcasts.google.com/feed/aHR0cHM6Ly9mZWVkcy5zaW1wbGVjYXN0LmNvbS9PQjVGa0lsOA/episode/OWVmNjk3M2QtNjY2Ni00MmYyLTg4MTQtZWE3MDZiYjU2N2Rk?sa=X&amp;ved=0CAUQkfYCahcKEwj4u96wo8XzAhUAAAAAHQAAAAAQAQ" rel="noopener nofollow ugc">the PyTorch Developer Podcast</a>.</p>
<p>I asked a similar question <a href="https://ai.stackexchange.com/q/32033/38532" rel="noopener nofollow ugc">elsewhere</a> but it was the wrong forum.</p>

ptrblck: <p>Yes, you can depend on runtime errors and as long as no errors are raised, you code should be fine.<br>
One difference would be that you are not allowed to set the <code>requires_grad</code> attribute on tensors from an <code>inference_mode</code> context:</p>
<pre><code class="lang-python">with torch.no_grad():
    x = torch.randn(1)
    y = x + 1

y.requires_grad = True
z = y + 1
print(z.grad_fn)
&gt; &lt;AddBackward0 object at 0x7fe9c6eafdf0&gt;

with torch.inference_mode():
    x = torch.randn(1)
    y = x + 1

y.requires_grad = True
&gt; RuntimeError: Setting requires_grad=True on inference tensor outside InferenceMode is not allowed.
</code></pre>

Sakshi_Bhatia: <p>Woah that’s interesting! So if I have a dummy tensor to ensure compatible dimensions inside my model tensor, I should preferably set that dummy to .inference_mode() since I do not need to take gradients on that. Am I right?</p>
Best practice for testing a pre-trained model without data leakage?

mehdiamlal: <p>Hi everyone,</p>
<p>I’m looking for the best practice on how to test a fine-tuned model loaded from HF Hub properly.</p>
<p>Specifically, I’m using the <code>cm93/resnet18-eurosat</code> model, which was fine-tuned on the EuroSAT dataset. My goal is to verify its performance on my own machine.</p>
<p>My core concern is <strong>data leakage</strong>. I know I shouldn’t test the model on data it was trained on. However, if I load the original dataset and create a test split, how can I be certain that these test samples weren’t already used during the original fine-tuning process by the author? I’m worried I might accidentally get an inflated score that doesn’t reflect real-world performance.</p>
<p>What is the standard, industry-accepted procedure here?</p>
<p>Thanks in advance!</p>

ptrblck: <p>The dataset is split into training, validation, and test in the <a href="https://huggingface.co/datasets/cm93/eurosat">HF repo</a> and I would assume the author of the pretrained model would respect these dataset splits.</p>
<p>I’m not familiar with a method verifying if no data leak happened.</p>

mehdiamlal: <p>So would I just test it on the test set, assuming the model was not trained on them? I apologise for the trivial question, I’m relatively new to the workflow.</p>

ptrblck: <p>Yes, you could test the model (without training) on the test set and should see the same reported test metric.</p>
How to use NestedTensor in C++?

byrzhm: <p>I want to use NestedTensor to group a bunch of small tensors for faster execution. I’m running into a problem while prototyping my project.</p>
<p>I want to print a NestedTensor in C++. Here’s my C++ code:</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">void print_nested_tensor(const at::Tensor&amp; tensor) {
  auto ptr = at::native::get_nested_tensor_impl(tensor);
  auto buffer = ptr-&gt;get_buffer();
  float* data_ptr = buffer.data_ptr&lt;float&gt;();
  for (int i = 0; i &lt; buffer.numel(); ++i) {
    std::cout &lt;&lt; "Data: " &lt;&lt; data_ptr[i] &lt;&lt; ", ";
  }
}

PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
  m.def("print_nested_tensor", &amp;print_nested_tensor, "Print nested tensor");
}
</code></pre>
<p>And here’s my Python test script:</p>
<pre data-code-wrap="python"><code class="lang-python">from cpp_extension import *
import torch
from torch.nested import nested_tensor, as_nested_tensor

a = torch.randn(2, 3, requires_grad=False)
b = torch.randn(6, 3, requires_grad=True)
nt = as_nested_tensor([a, b], layout=torch.jagged, dtype=torch.float32)

print_nested_tensor(nt)
</code></pre>
<p>But segmentation fault <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_tear.png?v=14" title=":smiling_face_with_tear:" class="emoji" alt=":smiling_face_with_tear:" loading="lazy" width="20" height="20"></p>
<pre><code class="lang-auto">[1]    2734946 segmentation fault (core dumped)  python test.py
</code></pre>

byrzhm: <p>I think the problem is in <code>auto buffer = ptr-&gt;get_buffer(); </code></p>

byrzhm: <p>I don’t know why, but it works.</p>
<p>Change</p>
<pre data-code-wrap="python"><code class="lang-python">layout=torch.jagged
</code></pre>
<p>to</p>
<pre data-code-wrap="python"><code class="lang-python">layout=torch.strided
</code></pre>
NVIDIA GeForce RTX 5070 Ti with CUDA capability sm_120 is not compatible with the current PyTorch installation.

Sushant_Singh: <p>CUDA available: True<br>
C:\Users\Admin\AppData\Local\Programs\Python\Python310\lib\site-packages\torch\cuda_<em>init</em>_.py:287: UserWarning:<br>
NVIDIA GeForce RTX 5070 Ti with CUDA capability sm_120 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_61 sm_70 sm_75 sm_80 sm_86 sm_90.<br>
If you want to use the NVIDIA GeForce RTX 5070 Ti GPU with PyTorch, please check the instructions at <a href="https://pytorch.org/get-started/locally/" rel="noopener nofollow ugc">https://pytorch.org/get-started/locally/</a></p>

ptrblck: <p>Install any of our stable or nightly binaries with CUDA 12.8 which is required for Blackwell GPUs.</p>
Pytorch equivalent of keras for FCN time series classification model

stomar: <p>I am new to pytorch, I am trying to port this tensorflow code of implementation of FCN model (<a href="https://github.com/hfawaz/dl-4-tsc/blob/master/classifiers/fcn.py" class="inline-onebox" rel="noopener nofollow ugc">dl-4-tsc/classifiers/fcn.py at master · hfawaz/dl-4-tsc · GitHub</a>) for time series classification problem to pytorch.</p>
<pre><code class="lang-auto">class FCNBlock(nn.Module):
  def __init__(self, in_channels, out_channels, kernel_size):
    super().__init__()

    self.conv = nn.Conv1d(in_channels, out_channels, kernel_size)
    self.bn = nn.BatchNorm1d(out_channels)
    self.relu = nn.ReLU()

  def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
    return self.relu(self.bn(self.conv(x)))


class FCN(nn.Module):
  def __init__(self, in_channels, num_classes, layers=[128, 256, 128], kss=[7, 5, 3]):
    super().__init__()

    self.layer_1 = FCNBlock(in_channels, layers[0], kss[0])
    self.layer_2 = FCNBlock(layers[0], layers[1], kss[1])
    self.layer_3 = FCNBlock(layers[1], layers[2], kss[2])
    self.pool = nn.AvgPool1d(1)
    self.flatten = nn.Flatten()
    self.fc = nn.Linear(layers[2], num_classes)

  def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
    x = self.convblock1(x)
    x = self.convblock2(x)
    x = self.convblock3(x)
    x = self.gap(x)
    x = self.flatten(x)
    x = self.fc(x)
    return x
</code></pre>
<p>However, my confusion is - in keras there is an input and output layer but in pytorch we start with the conv blocks. Also, the softmax activation after the average pooling layer, how to include that in the model definition? Is there any other discrepancies in my implementation that doesn’t match with the tensorflow code? Please advise.</p>
RuntimeError when exponentiating matrix using eigendecomposition

curious-droid: <p>I wish to use autograd to track gradients through a matrix exponentiation. I initially tried to use <code>torch.linalg.matrix_exp</code>, but it runs too slowly for my purposes, so instead, I am exponentiating the matrix by eigendecomposition (diagonalizing and raising the diagonal matrix to the appropriate power).</p>
<pre><code class="lang-auto">
        Lambda, V = torch.linalg.eig(self.Xi)
        V = V.to(dtype=torch.cdouble)
        Lambda = Lambda.to(dtype=torch.cdouble)
        V_inv = torch.linalg.inv(V)

        exp_Lambda_t = torch.diag_embed(torch.exp(Lambda.unsqueeze(0) * t.unsqueeze(1))) 
        
        z0_c = self.z0.to(dtype=torch.cdouble).unsqueeze(-1) 

        V_inv_z0 = V_inv @ z0_c 
        V_inv_z0 = V_inv_z0.unsqueeze(0).expand(len(t), -1, -1) 

        z_t = torch.matmul(V, torch.matmul(exp_Lambda_t, V_inv_z0))
        z_t = z_t.squeeze(-1)
        
        z_t=torch.abs(z_t)
</code></pre>
<p>Theoretically, the result of this computation should be a real matrix that does not depend on the phase of the chosen eigenvectors, but PyTorch throws <code>RuntimeError: linalg_eig_backward: The eigenvectors in the complex case are specified up to multiplication by e^{i phi}. The specified loss function depends on this quantity, so it is ill-defined.</code>Interestingly, this error only happens sporadically.</p>
<p>Printing out the first row of the offending matrix <code>z_t</code>, we see that the entries are not actually fully real, and due to some floating-point imprecision, it has nonzero imaginary parts. I tried to sidestep this issue by taking the complex norm of each entry, but the <code>RuntimeError</code> remained.</p>
<pre><code class="lang-auto">[-49.61000061-1.98751675e-15j  -1.63878882-1.43775610e-14j
   -0.65327233-5.49382679e-15j  -0.26159999+2.13188759e-15j
    1.57380724-7.87235174e-16j  -0.06189372-4.70826694e-15j]
</code></pre>
<p>Is there any way to reduce the floating-point imprecision, or sidestep the error thrown by Pytorch and force it to continue? Or are there better matrix exponentiation methods that I should use instead? I have also been seriously considering downgrading to <code>torch&lt;=1.10</code> to avoid the error.</p>
<p>Thanks in advance.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/curious-droid">@curious-droid</a>,</p>
<p>I’d assume the gradient through this operation, which is dependent upon calling<code>torch.linalg.eig</code>, is ill-defined as the gradients of its eigenvectors are only well defined when the corresponding eigenvalues are distinct (see the docs for more information on this: <a href="https://docs.pytorch.org/docs/stable/generated/torch.linalg.eig.html" class="inline-onebox" rel="noopener nofollow ugc">torch.linalg.eig — PyTorch 2.7 documentation</a> )</p>
<p>If your eigenvalues aren’t distinct, then it won’t have a well-defined gradient.</p>
Torch compile comsumes very high gpu memory

wadewang: <p>I set `flag_do_torch_compile` to True in <a href="https://github.com/KwaiVGI/LivePortrait/blob/main/src/config/argument_config.py" class="inline-onebox" rel="noopener nofollow ugc">LivePortrait/src/config/argument_config.py at main · KwaiVGI/LivePortrait · GitHub</a> to accelarate model inference, but I found the gpu memory is very high (about 17GB) even the model inference is finished, compared to the gpu memory (about 2.5GB) when disable torch compile.</p>
<p>My machine is cloud server with Nvidia A30 and ubuntu 20</p>