akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
PyTorch Image Models (timm)

username: hikmat_farhat
cooked: <p>I am getting very different results when using pretrained models using torchvision.models and timm.<br>
For example the only difference in my code is between</p>
<pre><code class="lang-auto">self.model=models.resnet18(weights=models.ResNet18_Weights.DEFAULT)
 self.model.fc=nn.Linear(self.model.fc.in_features,num_classes)
return self.model
</code></pre>
<p>and</p>
<pre><code class="lang-auto">self.model=timm.create_model(model_name='resnet18',num_classes=num_classes,pretrained=True)
return self.model
</code></pre>
<p>The results are show below for the training accuracy (it is similar for validation accuracy but I am allowed to post one figure).</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f2ff5b334af41a8df2dedabee3ad17137708a30e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f2ff5b334af41a8df2dedabee3ad17137708a30e" title="train_acc VS epoch"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2ff5b334af41a8df2dedabee3ad17137708a30e_2_690x345.jpeg" alt="train_acc VS epoch" data-base62-sha1="yFEE3Ak5rD6bpKnrpDJhZ1WNHb0" width="690" height="345" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2ff5b334af41a8df2dedabee3ad17137708a30e_2_690x345.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2ff5b334af41a8df2dedabee3ad17137708a30e_2_1035x517.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/2/f2ff5b334af41a8df2dedabee3ad17137708a30e.jpeg 2x" data-dominant-color="FDFDFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">train_acc VS epoch</span><span class="informations">1200×600 62.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: hikmat_farhat
cooked: <p>The problem occurs when the code runs on Kaggle. Locally it is fine</p>

username: Bjorn_Lindqvist
cooked: <p>Perhaps this post might be useful to you: <a href="https://blog.problemsolversguild.com/posts/2022-05-02-zero_init_last_resnetperformance.html" rel="noopener nofollow ugc">Exploring Timm vs Torchvision Resnet18 Difference</a> There are significant differences between torchvision’s and timm’s implementation of ResNet18.</p>
Join the PyTorch Docathon in June 2025!

username: svekars
cooked: <p>Come join us for the PyTorch Docathon: Jun 3 - 18, 2025!</p>
<p>Calling all PyTorch enthusiasts! Whether you’re a seasoned developer or just starting your journey with this powerful framework, we invite you to join our upcoming docathon. This is a fantastic opportunity to contribute to the project, make a real impact, and connect with like-minded individuals in the community.</p>
<h2><a name="p-470183-whats-on-the-agenda-1" class="anchor" href="#p-470183-whats-on-the-agenda-1"></a>What’s on the agenda?</h2>
<p>We will work on various documentation tasks in <a href="https://github.com/pytorch/pytorch" rel="noopener nofollow ugc">pytorch/pytorch</a> and <a href="https://github.com/pytorch/tutorials" rel="noopener nofollow ugc">pytorch/tutorials</a> repositories. More details will be provided on the day of the docathon. Make sure to RSVP to receive all the notifications about the event.</p>
<h2><a name="p-470183-why-participe-2" class="anchor" href="#p-470183-why-participe-2"></a>Why Participe?</h2>
<ul>
<li><img src="https://discuss.pytorch.org/images/emoji/apple/writing_hand.png?v=14" title=":writing_hand:" class="emoji" alt=":writing_hand:" loading="lazy" width="20" height="20"> Contribute: Help improve PyTorch documentation and make a lasting impact.</li>
<li><img src="https://discuss.pytorch.org/images/emoji/apple/light_bulb.png?v=14" title=":light_bulb:" class="emoji" alt=":light_bulb:" loading="lazy" width="20" height="20">Learn: Gain insights and knowledge from fellow developers.</li>
<li><img src="https://discuss.pytorch.org/images/emoji/apple/handshake.png?v=14" title=":handshake:" class="emoji" alt=":handshake:" loading="lazy" width="20" height="20">Connect: Meet other PyTorch enthusiasts and expand your network.</li>
</ul>
<p>Ready to join the fun? RSVP now and let us know you’re coming! <a href="https://community.linuxfoundation.org/events/details/lfhq-pytorch-foundation-presents-pytorch-docathon-june-3rd-18th-2025/" rel="noopener nofollow ugc">Click here to register</a>.</p>

username: ptrblck
action_code: pinned_globally.enabled

username: hejie
cooked: <p>good, <strong>thanks for contributing!</strong></p>
Custom Fold and Unfold

username: AnomUser8765
cooked: <pre><code class="lang-auto">def flatten(inputs: torch.Tensor, start_dim: int = 0, end_dim: int = -1) -&gt; torch.Tensor:
    """
    Flattens a tensor from start_dim to the end.
    """
    # TODO
    n_dim = len(inputs.shape)
    if start_dim &gt; n_dim:
        raise ValueError
    elif start_dim == n_dim:
        return inputs
    
    return inputs.view(*inputs.shape[: start_dim], -1, *inputs.shape[end_dim + 1:])


def unfold2d(inputs: torch.Tensor, kernel_size: int, stride: int = 1, padding: int = 0) -&gt; torch.Tensor:
    """
    Performs a 2D unfold operation.
    """
    # TODO
    N, C, H, W = inputs.shape
    # Pad
    if padding &gt; 0:
        padded_inputs = torch.zeros((N, C, H + 2*padding, W + 2*padding), device=inputs.device)
        padded_inputs[:, :, padding:padding + H, padding:padding + W] = inputs
        W_pad = W + 2*padding
        H_pad = H + 2*padding
    else:
        padded_inputs = inputs
        W_pad = W
        H_pad = H
    
    # Unfold
    h_new = int((H_pad - kernel_size)/stride) + 1
    w_new = int((W_pad - kernel_size)/stride) + 1

    patches = []
    for i in range(0, h_new - kernel_size + 1, stride):
        for j in range(0, w_new - kernel_size + 1, stride):
            patch = padded_inputs[:, :, i:i + kernel_size, j:j + kernel_size] # N, C, kernel_size, kernel_size
            patch.view(N, C*kernel_size*kernel_size) # N, C*kernel_size*kernel_size
            patches.append(patch) 
    patches = torch.stack(patches, dim=2) # (N, C*kernel_size*kernel_size, h_new, w_new)

    return patches

    

def fold2d(inputs: torch.Tensor, output_size: tuple[int, int], kernel_size: int, stride: int = 1, padding: int = 0) -&gt; torch.Tensor:
    """
    Performs a 2D fold operation.
    """
    # TODO
    N, C_k_k, L = inputs.shape
    C = int(C_k_k / (kernel_size * kernel_size))
    H, W = output_size
    h_pad, w_pad = H + 2*padding, W + 2*padding
    unfolded = torch.zeros((N, C, h_pad, w_pad), device=inputs.device)
    normalizator = torch.zeros_like(unfolded)
    
    # Unfold
    indx = 0
    for i in range(0, h_pad - kernel_size + 1, stride):
        for j in range(0, w_pad - kernel_size + 1, stride):
            patch = inputs[:, :, indx].view(N, C, kernel_size, kernel_size) # Tomamoms el patch indx y lo reordenamos N, C, kernel_size, kernel_size
            unfolded[:, :, i:i + kernel_size, j:j + kernel_size] += patch
            normalizator[:, :, i:i + kernel_size, j:j + kernel_size] += 1
            indx += 1
    # Remove padding
    if padding &gt; 0:
        output = unfolded[:, :, padding:padding + H, padding:padding + W]
        normalizator = normalizator[:, :, padding:padding + H, padding:padding + W]
    else:
        output = unfolded
        normalizator = normalizator
    
    # Normalize
    output = output / normalizator.clamp(min=1)
</code></pre>
"Unknown type name": forward reference breaking torchscript

username: rrm39
cooked: <p>I am trying to script a  model with the <code>zuko</code> package (<a href="https://github.com/probabilists/zuko" class="inline-onebox" rel="noopener nofollow ugc">GitHub - probabilists/zuko: Normalizing flows in PyTorch</a>). I instantiate a model as<br>
<code>flow = zuko.flows.MAF(features=1, context=1, transforms=4, hidden_features=[32,32])</code></p>
<p>then after training, I script it as<br>
<code>sm = torch.jit.script(flow) </code></p>
<p>which results in the error</p>
<pre><code class="lang-auto">Unknown type name 'LazyTransform':
  File "/global/homes/r/rmastand/.local/lib/python3.8/site-packages/zuko/lazy.py", line 74
    @property
    def inv(self) -&gt; LazyTransform:
                     ~~~~~~~~~~~~~ &lt;--- HERE
        r"""A lazy inverse transformation :math:`x = f^{-1}(y | c)`."""
</code></pre>
<p>The issue comes from [this file] (<a href="https://github.com/probabilists/zuko/blob/313481e0873b4c98636314d6c3feaea30284b28a/zuko/lazy.py#L80" class="inline-onebox" rel="noopener nofollow ugc">zuko/zuko/lazy.py at 313481e0873b4c98636314d6c3feaea30284b28a · probabilists/zuko · GitHub</a>) — <code>LazyTransform</code> is not defined until line 80, but it is forward-referred to on line 74. Is there a decorator that I can attach to these functions? The entire repository seems based around the <code>LazyTransform</code>, so I don’t think I can avoid compiling it.</p>
Masked AutoEncoder Pre-training

username: samuel3
cooked: <p>Hi everyone,</p>
<p>A few questions here.</p>
<p>At what epoch do you notice that masked autoencoders start producing similar-looking reconstructions during the pretraining phase? I’ve experimented on pretraining the BTCV dataset and notice that after ~800 epochs, the results are really no different than the first 20 or so.</p>
<p>My loss never really drops before 0.9 either.  I’m using a low learning rate and weight decay following the hyperparameters of the original paper.  Is this common?</p>
<p>Also, it best to only move to the downstream task (in my case segmentation) once the reconstructions look decent?</p>

username: gayalkuruppu
cooked: <p>Were you able to solve this issue? I also faced the same issue (celeba dataset), tried both <a href="https://github.com/facebookresearch/mae/blob/main/models_mae.py" class="inline-onebox" rel="noopener nofollow ugc">mae/models_mae.py at main · facebookresearch/mae · GitHub</a> and <a href="https://github.com/lucidrains/vit-pytorch/blob/main/vit_pytorch/mae.py" class="inline-onebox" rel="noopener nofollow ugc">vit-pytorch/vit_pytorch/mae.py at main · lucidrains/vit-pytorch · GitHub</a>. Appreciate any leads.</p>
Is my loss function normal?

username: hongshengqi
cooked: <p>my loss changes along iteration as the figure.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/1/815b33ced51073a7a0c09ac170806f948bb030cc.png" data-download-href="https://discuss.pytorch.org/uploads/default/815b33ced51073a7a0c09ac170806f948bb030cc" title="2025-05-05 10-32-20屏幕截图"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/815b33ced51073a7a0c09ac170806f948bb030cc_2_640x500.png" alt="2025-05-05 10-32-20屏幕截图" data-base62-sha1="iskZCsa96zNO4ZDkxRZWAqBAqTy" width="640" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/1/815b33ced51073a7a0c09ac170806f948bb030cc_2_640x500.png, https://discuss.pytorch.org/uploads/default/original/3X/8/1/815b33ced51073a7a0c09ac170806f948bb030cc.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/1/815b33ced51073a7a0c09ac170806f948bb030cc.png 2x" data-dominant-color="E9F1F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">2025-05-05 10-32-20屏幕截图</span><span class="informations">918×717 87.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is my loss normal?</p>
<p>I use “optimizer = optim.SGD(parameters, lr = args.learning_rate, weight_decay = args.weight_decay_optimizer)”, and I train three standalone models simultaneously (the loss depends on all three models dont share any parameters).</p>
<p>Why my loss trend differs from the curves at many papers which decrease in a stable manner?</p>

username: J_Johnson
cooked: <p>Help me out. Is the horizontal axis per epoch or per batch?</p>
<p>If per batch(highly likely), that just shows that some batches contribute more loss while others less, progressively getting better.</p>
<p>If per epoch, I’m not sure, that would be an odd chart - perhaps LR is set too high? Definitely going to have over-fitting with 10k+ epochs(and a pricey power bill).</p>
Should I use torchmetrics or sklearn.metrics

username: Igor_Santarek
cooked: <p>Hi, should I use torchmetrics MulticlassF1Score or skicit-learn sklearn.metrics.f1_score for calculating F1 score of my classification model? I’m a beginner and I want to know what should I use to be professional. Both seem to be the same and I don’t know what role does one solve over another.</p>

username: ptrblck
cooked: <p><code>torchmetrics</code> should be able to accept PyTorch tensors so might be more convenient as you would be able to pass e.g. <code>CUDATensor</code>s directly to their methods.</p>
Pytorch save execution without NaN values

username: Johannes_Vogt
cooked: <p>Hey,</p>
<p>there is the possibility to check and replace Nan values in Pytorch using torch.isnan and torch.nan_to_num.</p>
<p>For larger models, one could always loop over all parameters to replace the NaN (and Inf) values.</p>
<p>Is there an option to automate these operations and replace NaN values directly when they are created (e.g. in a matrix multiplication)?</p>

username: ptrblck
cooked: <p>I haven’t seen an automated approach replacing NaNs but it would also be interesting to learn about your use case. E.g. replacing NaNs in the forward would not fix the reason these were created.</p>
Is cuda installed automatically when following pytorch instlalation directions?

username: Almenon
cooked: <p>I installed pytorch but than ran into a “OSError: CUDA_HOME environment variable is not set” error when I tried installing flash attention after that. I’m curious if the cuda toolkit is installed automatically when following pytorch installation steps in <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a>. If it is installed, where it is installed?</p>
<p>If it’s not installed, some feedback is it would be helpful to mention that you need to install it separately.</p>
<p>Thanks for making pytorch!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Almenon" data-post="1" data-topic="215493">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/almenon/48/75137_2.png" class="avatar"> Almenon:</div>
<blockquote>
<p>I’m curious if the cuda toolkit is installed automatically when following pytorch installation steps</p>
</blockquote>
</aside>
<p>The PyTorch binaries ship with their own CUDA <em>runtime</em> dependencies, but do not install a full CUDA toolkit (for development).<br>
Your <code>flash-attn</code> build seems to fail as it’s expecting a full CUDA compiler to build this package from source, as the package ships with a <a href="https://pypi.org/project/flash-attn/2.7.3/#files">source distribution</a> only (not wheels).</p>

username: Almenon
cooked: <p>Great answer, thanks! Unfortunately compiling flash-attn is taking <em>forever</em> (more than 3 hours!) so I’m going to have to switch tactics. I’m thinking I’ll switch from windows to WSL.</p>

username: Guillermo_Barbadillo
cooked: <p>On Ubuntu, I have found that the requirement to set the <code>CUDA_HOME</code> depends on the <code>flash-attn</code> version. I have been able to install it with <code>pip install flash-attn==2.6.3</code>, but newer version such as <code>2.7.2</code> raise that problem</p>
How to perform distributed communication (NCCL) using LibTorch?

username: Jiawen_Niu
cooked: <p>For example, in the Python API, there are interfaces like torch.distributed.allreduce, but I couldn’t find similar interfaces in the LibTorch C++ API.</p>
<p>Thank you for all the responses.</p>

username: fduwjj
cooked: <p>In cpp, there are APIs inside ProcessGroupNCCL which directly calls into nccl API, is this something you are looking for?</p>

username: Jiawen_Niu
cooked: <p>Yes, I saw it. Thank you very much for your reply.</p>

username: Jiawen_Niu
cooked: <p>Hope to consult another question: How to assign a specific stream for ProcessGroupNCCL and subsequent communication operations?</p>
Question about default allowed globals

username: cwindolf
cooked: <p>I’m interested in checking that I’m taking the right approach to storing and loading models with <code>torch.save</code>/<code>load</code> using the newish default <code>weights_only=True</code>. I’m distributing code which fits, saves, and loads a few models, and it makes sense to me that I would need to register my own <code>nn.Module</code> subclasses with <code>torch.serialization.add_safe_globals</code>.</p>
<p>I’m a bit more confused about torch’s own classes. For instance, I have to add <code>nn.Linear</code>, <code>nn.ReLU</code>, etc to the safe globals list in order for saving and loading to work. Since these are fundamental classes in torch, this makes me wonder if I’m cutting corners or doing something that wasn’t intended. Am I introducing a security risk to my users by doing this, maybe in a scenario involving some monkey patching of these classes? Should I just use weights_only=False? Should I be doing everything with state dicts instead? If this is the intended usage of this feature, I’m curious to understand why, if anyone can offer guidance.</p>
<p>Thanks!!</p>
Compatibility issue between FramePack and RTX 5090 – CUDA Error

username: Bernard_Vincent
cooked: <p>Hello everyone,</p>
<p>I’m currently experiencing an issue trying to run FramePack on my system equipped with an RTX 5090. Despite installing the latest PyTorch nightly build (2.8.0.dev20250501+cu128) and CUDA Toolkit 12.8, I encounter the following error during execution:</p>
<p>vbnet</p>
<p>CopierModifier</p>
<pre><code class="lang-auto">RuntimeError: CUDA error: no kernel image is available for execution on the device
</code></pre>
<p>I’ve tried several solutions, including updating NVIDIA drivers and reinstalling PyTorch with the appropriate options, but the issue persists.</p>
<p><strong>My setup:</strong></p>
<ul>
<li>GPU: NVIDIA RTX 5090</li>
<li>OS: Windows 11 Pro</li>
<li>Python: 3.10.11</li>
<li>CUDA Toolkit: 12.8</li>
<li>PyTorch: 2.8.0.dev20250501+cu128</li>
</ul>
<p>I’m aware that the RTX 50 series is relatively new and compatibility issues might occur. If anyone has encountered a similar problem or has suggestions to resolve this error, I’d really appreciate your help.</p>
<p>Thanks in advance for your support!</p>

username: ptrblck
cooked: <p>I don’t know what <code>FramePAck</code> is but our binaries built with CUDA 12.8 work on Blackwell architectures. You could create a new and empty virtual environment, install the latest stable or nightly binary with CUDA 12.8 support, and execute a simple smoke test without installing <code>FramePack</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print(torch.version.cuda)
print(torch.cuda.is_available())
print(torch.cuda.get_arch_list())
print(torch.randn(1).cuda())
</code></pre>

username: Bernard_Vincent
cooked: <p>Hi [<a class="mention" href="/u/ptrblck">@ptrblck</a>],</p>
<p>I just wanted to sincerely thank you for the valuable help you provided regarding the installation of <code>flash-attn</code>. Your suggestion to use the <code>--no-build-isolation</code> flag was exactly what I needed — it allowed <code>pip</code> to properly detect <code>torch</code> during the build process, which was the root cause of all my installation issues.</p>
<p>Thanks to your advice, I was finally able to compile and install <code>flash-attn</code> correctly in a clean virtual environment with CUDA 12.6 and Python 3.10, without dependency conflicts or broken packages.</p>
<p>The only remaining issue I’m now working on is saving the final output video. It currently fails due to a missing module (<code>PyAV</code>), but I’m now installing <code>av</code> to fix that.</p>
<p>Thanks again for your time and clear explanation. Your help really made a difference. <img src="https://discuss.pytorch.org/images/emoji/apple/folded_hands.png?v=14" title=":folded_hands:" class="emoji" alt=":folded_hands:" loading="lazy" width="20" height="20"></p>
<p>Best regards,<br>
Bernard (aka test5090 <img src="https://discuss.pytorch.org/images/emoji/apple/grinning_face_with_smiling_eyes.png?v=14" title=":grinning_face_with_smiling_eyes:" class="emoji" alt=":grinning_face_with_smiling_eyes:" loading="lazy" width="20" height="20">)</p>
How to Overcome EEG Data Overfitting in Deep Learning Models?

username: xenablgn
cooked: <p>Hi everyone,</p>
<p>I’m working on a deep learning model to analyze 4D EEG data (Batch × Frequency × Channels × Time), where the input is trial-wise power percentage values normalized relative to baseline. The trials come from multiple subjects, and I treat them as independent samples. I use stratified shuffling to split the data into training, validation, and test sets.</p>
<p>I’ve experimented with several architectures. The best-performing one so far is based on a Temporal Convolutional Network (using a third-party PyTorch implementation), which outperformed Conv1D+LSTM stacks in capturing single-trial EEG dynamics. However, I’m consistently running into overfitting issues.</p>
<p>I’ve tried a range of regularization and augmentation techniques:</p>
<ul>
<li>Dropout, weight decay, and weight perturbations</li>
<li>Data augmentations like Gaussian noise, time masking, and other published methods</li>
<li>I even attempted synthetic data generation via GANs, but the generator performance was poor and didn’t help much</li>
</ul>
<p>Despite these efforts, my model still overfits. Has anyone faced similar challenges when working with trial-wise EEG power data? I’d really appreciate any advice on:</p>
<ul>
<li>Architectures that worked well for you</li>
<li>Data handling strategies (e.g., subject-wise splits, other normalization approaches)</li>
<li>Any augmentation techniques you found particularly helpful</li>
</ul>
<p>I’m open to any suggestions. Honestly, I’m a bit stuck and would really value the community’s input.</p>
<p>Thanks in advance!</p>
Install pytorch cuda 11.4 with conda

username: Helena099
cooked: <p>Hello,<br>
I tried to download PyTorch that matches my device:<br>
Driver Version: 470.199.02<br>
CUDA Version: 11.4<br>
Ubuntu: 20.04.4<br>
However, I cannot find the Pytorch version for CUDA 11.4 on the PyTorch official website. Most of the information related to CUDA 11.4 on the internet is kind of outdated.</p>
<p>Also, I tried torch==2.5.1, I tried to run torch.cuda.is_available() and the output is False. And the output of torch.cuda.device_count() is 1. Anyone has the idea of how to fix this problem? Thank you!!</p>

username: ptrblck
cooked: <p>I would not recommend trying to install such old PyTorch releases. Instead try to install the latest one with CUDA 11.8 or update your NVIDIA driver and install the latest PyTorch binary with any CUDA 12 runtime dependency.</p>
How to input CNN's output into LSTM?

username: hirundo
cooked: <p>Hello.<br>
I am trying to use CNN-LSTM for the audio data. (e.g. spectrograms image)<br>
The output shape of my CNN model is [batch_size, features] and LSTM expect an input of [batch_size, seq_len, features].</p>
<p>My question is how to make seq_len from CNN’s output.</p>
<p>Also, I want to study CNN-LSTM, so if you have any good materials, please share them <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: vdw
cooked: <p>I assume the input for your CNN – i.e., a current batch – contains all the spectrogram images to form the time series you want to give the LSTM as input. Otherwise I wouldn’t know where your sequence comes from.</p>
<p>In this case the <code>batch_size</code> of the CNN becomes the <code>seq_len</code> for the LSTM, and for the latter, the <code>batch_size</code> is just 1 (as you don’t have batches with multiple sequences).</p>

username: hirundo
cooked: <p>Thank you for quick response !</p>
Profiling libtorch code. What is _init(), and why is it hogging my CPU?

username: GlenH
cooked: <p>Hi all,</p>
<p>I’m using libtorch (C++) for a non-typical use case. I need it to do some massively parallel dynamics computations. I know this isn’t the intended use case, but I have reasons.</p>
<p>In any case, the code is fairly slow and I’m trying to speed it up as much as possible. I’ve written some test code that just calls my dynamics routine thousands of times in a for() loop. However, I don’t understand the results I’m getting from gprof. Specifically, gprof reports that fully half my time is spent inside “_init” (25 seconds of a 50 second run time).</p>
<p>I know C++ used to use _init during the initialization of libraries, but it’s been deprecated for ages. Does lib torch still use _init, and if so are there any steps I can take to reduce the overhead it’s consuming?</p>
I install torchvision from source, but I see RuntimeError: operator torchvision::nms does not exist

username: George_Polya
cooked: <h1><a name="p-470057-scenario-1-install-pytorch-by-pip-1" class="anchor" href="#p-470057-scenario-1-install-pytorch-by-pip-1"></a>Scenario 1. install pytorch by pip</h1>
<ol>
<li>install pytorch</li>
</ol>
<pre><code class="lang-auto">pip install torch==2.5.0 --index-url https://download.pytorch.org/whl/cu124 
</code></pre>
<ol start="2">
<li>install torchvision from source</li>
</ol>
<pre><code class="lang-auto">wget -O vision.zip https://github.com/pytorch/vision/archive/refs/tags/v0.20.0.zip &amp;&amp; unzip vision.zip
cd vision-0.20.0
pip install . 
</code></pre>
<ol start="3">
<li>I see RuntimeError: operator torchvision::nms does not exist</li>
</ol>
<h1><a name="p-470057-scenario-2-install-pytorch-from-source-2" class="anchor" href="#p-470057-scenario-2-install-pytorch-from-source-2"></a>Scenario 2. install pytorch from source</h1>
<ol>
<li>install pytorch from source</li>
</ol>
<pre><code class="lang-auto">git clone -b v2.5.0 --recurse-submodule https://github.com/pytorch/pytorch.git pytorch
cd pytorch
python3 setup.py install
</code></pre>
<ol start="2">
<li>install torchvision from source</li>
</ol>
<pre><code class="lang-auto">wget -O vision.zip https://github.com/pytorch/vision/archive/refs/tags/v0.20.0.zip &amp;&amp; unzip vision.zip
cd vision-0.20.0
pip install . 
</code></pre>
<ol start="3">
<li>I see RuntimeError: operator torchvision::nms does not exist</li>
</ol>
<p>When I use <code>python3 setup.py install</code><br>
How to solve this error? I need to use torchvision from source.</p>
Recompilations leading to timeout

username: Naveen_Marri
cooked: <p>Hi, how do we deal with avoiding recompiles? I am trying to compile a model with dynamic input shapes and it triggers recompiles for every iteration and eventually timing out.</p>
<pre><code class="lang-auto">model = torch.compile(model, fullgraph=True, mode="reduce-overhead", dynamic=True)
</code></pre>
<p>recompilation log</p>
<pre><code class="lang-auto">[rank1]:V0501 19:01:33.888000 146171 site-packages/torch/_dynamo/guards.py:2791] [0/7] [__recompiles] Recompiling function forward in /home/jovyan/model.py:129
[rank1]:V0501 19:01:33.888000 146171 site-packages/torch/_dynamo/guards.py:2791] [0/7] [__recompiles]     triggered by the following guard failure(s):
[rank1]:V0501 19:01:33.888000 146171 site-packages/torch/_dynamo/guards.py:2791] [0/7] [__recompiles]     - 0/6: tensor 'L['batch']['model_inputs'].data['input_ids']' size mismatch at index 1. expected 67, actual 50
packages/torch/_dynamo/guards.py:2791] [0/7] [__recompiles] Recompiling function forward in /home/jovyan/model.py:129
</code></pre>
Creating a Tensor with a value that is already encoded in bfloat16

username: anantanurag
cooked: <p>Hi,<br>
I am very new to PyTorch. But I would like to use its features to convert between datatypes.</p>
<p>Suppose I have a hex number that is already encoded in bfloat16 = 0xABCD</p>
<p>torch.tensor([0xABCD], dType=bfloat16) → this seems to treat 0xABCD as a hex encoded version of the decimal number 0d7117.</p>
<p>How can I initialise a variable that is already encoded in bfloat16 format and then have PyTorch convert it to its decimal value?</p>
(torch.__version__) 2.7.0+cu126

username: Ghada_Almousa
cooked: <p>i have this version<br>
torch 2.7.0+cu126<br>
i want to install<br>
pytorch-geometric<br>
pytorch-sparse<br>
pytorch-cluster<br>
pytorch-spline. each time i install them i get not compatible with 2.7.0+cu126</p>

username: ptrblck
cooked: <p>It seems PyG and the other 3rd party libraries might not have released a new version compatible with PyTorch 2.7.0 so you might want to ask this question on their GitHub repository.</p>
PyTorch Tensor Parallel

username: Chenchao_Zhao
cooked: <p>Is torch implementation of DTensor materially different from the MegatronLM TP? In Megatron it’s recommended to set <code>CUDA_DEVICE_MAX_CONNECTIONS</code> to 1 to enable TP communication overlap. Is it also true for torch DTensor TP? I got an answer from torch titan issue but I hope to get more context about the differences between torch TP and Megatron TP. Thanks!</p>
MPS Deployment Target Mismatch

username: Olek
cooked: <p>Hi all. I am are building PyTorch from source with MPS support and am having a problem with the Deployment Target<br>
Some context:<br>
Im using LibTorch to run inference in C++.<br>
My minimum deployment target is currently 11.</p>
<p>My build works correctly on all systems 11 and up as expected for CPU usage, and I also have no problem checking for MPS availability on any system as well.<br>
The problem Im having right now is that inference with MPS only seems to work correctly on the same Mac OSX that PyTorch was built on. Eg, I am building on 15.1 and it works correctly on my own system, but If I attempt to use this build on MacOS 14 it will not work.<br>
Below is the error I see when trying this:</p>
<pre><code class="lang-auto">RuntimeError: Failed to created pipeline state object, error: Error Domain=AGXMetalG14X Code=3 "Function kernel_index_offsets_32 has a deployment target (0x00020007) which is incompatible with this OS (0x00020006)." UserInfo={NSLocalizedDescription=Function kernel_index_offsets_32 has a deployment target (0x00020007) which is incompatible with this OS (0x00020006).}
</code></pre>
<p>My question is, is it possible to build PyTorch with MPS so that it can be deployed on a MacOS different from the one built on and if so, how can this be done?<br>
Im already setting CMAKE arguments related to the deployment target, but maybe Im missing something critical.</p>
<pre><code class="lang-auto">CMAKE_OSX_DEPLOYMENT_TARGET

</code></pre>
<p>Any help appreciated, thanks all.</p>

username: Olek
cooked: <p>For anyone who finds this issue in the future, it looks like Ive managed to solve the problem. After digging around for a while I managed to figure out how these problem kernels are being built.<br>
In Metal.cmake:</p>
<pre><code class="lang-auto">function(metal_to_air SRC TARGET FLAGS)
    add_custom_command(COMMAND xcrun metal -c -mmacosx-version-min=12.3 ${SRC} -o ${TARGET} ${FLAGS} ${METAL_CFLAGS}
                       DEPENDS ${SRC}
                       OUTPUT ${TARGET}
                       COMMENT "Compiling ${SRC} to ${TARGET}"
                       VERBATIM)
endfunction()
</code></pre>
<p>I’ve added <code>-mmacosx-version-min=12.3</code>. Without this the deployment target is the OS you are building on (or SDK being built with?)<br>
This suits my needs, but I think it should be the case that when building PyTorch the normal CMAKE arguments for minimum deployment target should make their way here somehow. I might make a fork later of some actual fix, but for now this will do the trick for me.</p>
APL support in pytorch

username: Meghana_R_Prakash
cooked: <p>I was going through the pytorch source code and wanted to use APL for blas and lapack operation however in FindAPL.cmake there is a check for library in the binary directory<br>
“FIND_PATH(APL_BIN_DIR NAMES armpl_lp64.dll libarmpl_lp64.a PATHS ${APL_BIN_SEARCH_PATHS})”</p>
<p>I have installed arm performance library using the below link:</p>
<p><a href="https://learn.arm.com/install-guides/armpl/#:~:text=Windows,-On%20your%20Windows&amp;text=Double%20click%20to%20open%20this,terms%20of%20this%20License%20Agreement'.&amp;text=Click%20'Install'%20and%20then%20',Finish" class="inline-onebox" rel="noopener nofollow ugc">Arm Performance Libraries | Arm Learning Paths</a>’%20to%20complete%20the%20installation</p>
<p>I don’t see any libraries in bin directory. Is this particular patch written in pytorch for a specific APL version. Currently I am using 24.10 version of Arm performance library.<br>
It would be really helpful if someone gives an intuition on why this is happening.</p>

username: mhall119
cooked: <p>Hey there, what operating system are you doing this on?</p>

username: Meghana_R_Prakash
cooked: <p>I am doing this on Linux</p>

username: mhall119
cooked: <p>Okay, and did you install it using the manual instructions or using a DEB or RPM package?</p>
<p>Also, which distro and version of Linux are you using?</p>
Questions about Pytorch 2.0 and mps

username: tpk
cooked: <p>Recently Pytorch had announced Pytorch 2.0, which is awesome</p>
<ol>
<li>It canonicalizes 2000+ primitives to 250+ essential ops, and 750+ ATen ops. Which reduce the implementation code by at least about a half</li>
<li>It use TorchDynamo which improves graph acquisition time</li>
<li>It uses faster code generation through TorchInductor</li>
</ol>
<p>However, as my understanding goes mps support is in very early stage, and according to <a href="https://github.com/pytorch/pytorch/issues/77764" rel="noopener nofollow ugc">General MPS op coverage tracking issue</a> there seems to be a lot more Aten Ops to be implemented.</p>
<p>So did Pytorch 2.0 change any codebase or direction that had been going since Pytorch 1.12 (I meant is TorchDynamo compatible with current mps graph acquisition methods?)?</p>
<p>Also TorchInductor will compile into <a href="https://github.com/openai/triton" rel="noopener nofollow ugc">Triton</a>, which seems to only support CUDA (, and maybe ROCm in the future). If Pytorch 2.0 came along, how this going to affected the mps support?</p>
<p>Sorry if I misunderstand something, I just want to know the general direction. Thanks</p>

username: marksaroufim
cooked: <p>As of today the default backend for torch.compile on MPS supported devices is aot_eager so you’d do <code>torch.compile(..., backend="aot_eager")</code></p>
<p>Inductor support for MPS might eventually happen but it’s primarily dependent on Triton supporting MPS. Inductor codegens triton kernels which then can run on GPU like devices, right now the support in Triton is mostly NVIDIA GPU focused so . You can make the ask here <a href="https://github.com/openai/triton" class="inline-onebox" rel="noopener nofollow ugc">GitHub - openai/triton: Development repository for the Triton language and compiler</a></p>
<p>Also the current MPS implementation is an eager mode implementation so it’s not dependent on graph capturee but generally compiler and hardware authors like graphs because there’s a well established litterature on how to optimize them</p>

username: Jeff_Coggshall
cooked: <p>Any update on this? I looked around but still haven’t seen anything on MPS graph compilation? Should I abandon hope or are you aware of anyone working on this?</p>

username: marksaroufim
cooked: <p>Nothing to share yet unfortunately <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

username: marksaroufim
cooked: <p>Happy update, we have experimental support for torch.compile that directly code generates Metal kernels now <a href="https://github.com/pytorch/pytorch/blob/main/torch/_inductor/codegen/mps.py" class="inline-onebox">pytorch/torch/_inductor/codegen/mps.py at main · pytorch/pytorch · GitHub</a></p>