akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
How many patterns does torch compile match and replace for now?

ljh: <p>Hi, I’m trying to investigate how pattern matching works in torch compile.<br>
I tried search for “register_replacement” and “gen_register_replacement”, which seem to be the way of registering patterns for matching.<br>
But I only got a few results (i thought there would be a lot rules for matching).</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/search?q=repo%3Apytorch%2Fpytorch%20gen_register_replacement&amp;type=code">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" alt="" width="32" height="32">

      <a href="https://github.com/search?q=repo%3Apytorch%2Fpytorch%20gen_register_replacement&amp;type=code" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <img width="500" height="500" src="https://github.githubassets.com/assets/github-logo-55c5b9a1fe52.png" class="thumbnail onebox-avatar" alt="">

<h3><a href="https://github.com/search?q=repo%3Apytorch%2Fpytorch%20gen_register_replacement&amp;type=code" target="_blank" rel="noopener nofollow ugc">Build software better, together</a></h3>

  <p>GitHub is where people build software. More than 150 million people use GitHub to discover, fork, and contribute to over 420 million projects.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I want to know that is it really that few rules for matching?<br>
Or the pattern matching is mainly just about flash attention (as discussed in <a href="https://dev-discuss.pytorch.org/t/how-is-pattern-matching-in-inductor-fx-implemented/1720/4" class="inline-onebox" rel="noopener nofollow ugc">How is pattern matching in inductor/fx implemented? - #4 by youkaichao - compiler - PyTorch Developer Mailing List</a>)</p>
Clueless Newbie Seeking Clarification

Redshift: <p>Hello Warning - I a total beginner. Google Gemini is guiding me to install this file which appears to be unavailable: <a href="https://download.pytorch.org/whl/cu118/faiss_gpu-1.7.2-cp311-cp311-manylinux2014_x86_64.whl" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu118/faiss_gpu-1.7.2-cp311-cp311-manylinux2014_x86_64.whl</a>  However it is rather insistent that it is available and to keep trying etc or switch to Anaconda/Conda. My preference is to go to go with first choice if possible, but I wonder if that file is no longer available to download?  Appreciate any guidance   Thanks!</p>

ptrblck: <p>I’m not familiar with your use case and why <code>faiss</code> is needed, but from their <a href="https://github.com/facebookresearch/faiss?tab=readme-ov-file#installing">repository</a>:</p>
<blockquote>
<p>Faiss comes with precompiled libraries for Anaconda in Python, see <a href="https://anaconda.org/pytorch/faiss-cpu">faiss-cpu</a>, <a href="https://anaconda.org/pytorch/faiss-gpu">faiss-gpu</a> and <a href="https://anaconda.org/pytorch/faiss-gpu-cuvs">faiss-gpu-cuvs</a>….</p>
</blockquote>
Questions about onnx with aten::copy

ZongYingLi: <p>I’m trying to export a model to ONNX for use in C++, but I keep encountering the following error:<br>
<code>torch.onnx.errors.UnsupportedOperatorError: Exporting the operator 'aten::copy' to ONNX opset version 17 is not supported.</code><br>
I’ve tried changing the ONNX opset version, but the issue persists.<br>
I’d greatly appreciate any advice or help on how to resolve this problem.</p>

Vitali-Matteo: <p>Have you solved this issue? I am in your same situation.</p>

justinchuby: <p>Please use the dynamo=True option and try again.</p>
"Cannot allocate memory" for multinode training

jiachenzhu: <p>I found that when I run my code with more than one node on my cluster, I will get error like:<br>
<code>[rank0]:   File … line 66, in _launch</code><br>
<code>[rank0]:     self.pid = os.fork()</code><br>
<code>[rank0]:                ^^^^^^^^^</code><br>
<code>[rank0]: OSError: [Errno 12] Cannot allocate memory</code></p>
<p>in all ranks. However, when I run it with one node, I never have the same error. So I want to know how could I debug my cluster setup to understand what’s wrong with my cluster. Here is the minimal code to allways reproduce the issue:</p>
<pre data-code-wrap="python"><code class="lang-python">import os

import torch

import torchvision

def main():

local_rank = int(os.environ["LOCAL_RANK"])

rank = int(os.environ["RANK"])

world_size = int(os.environ["WORLD_SIZE"])



torch.cuda.set_device(local_rank)

torch.distributed.init_process_group("nccl", world_size=world_size, rank=rank)



print(f"Local rank: {local_rank}, Rank: {rank}, World size: {world_size}")



################################################################################################

transform = torchvision.transforms.Compose([

torchvision.transforms.Resize(256, interpolation=torchvision.transforms.InterpolationMode.BICUBIC),

torchvision.transforms.CenterCrop(224),

torchvision.transforms.ToTensor(),

torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),

    ])

dataset = torchvision.datasets.ImageFolder("...", transform=transform)

sampler = torch.utils.data.DistributedSampler(dataset, num_replicas=world_size, rank=rank, shuffle=True, seed=0, drop_last=True)

loader = torch.utils.data.DataLoader(dataset, batch_size=1, sampler=sampler, num_workers=16, drop_last=True)



model = torchvision.models.vit_b_16(weights=torchvision.models.ViT_B_16_Weights.IMAGENET1K_V1)

model.to(local_rank)

model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[local_rank])



for i, (images, labels) in enumerate(loader):

print(images.shape, labels.shape)

if i &gt; 10:

break



torch.distributed.barrier(device_ids=[local_rank])

torch.distributed.destroy_process_group()

if __name__ == "__main__":

main()
</code></pre>
<p>If I remove the DDP line or I use only one node (even with more GPUs), I will never have the error. Can anyone help?</p>

ptrblck: <p>Check if shared memory is limited and increase it if needed.</p>

Arunprakash-A: <p>I ran your code on a node with 2 GPUs (<code>--nproc_per_node=2</code>), and it executed without any issues</p>
PT2E quantization doesn't reduce the model size

vladimir_risojevic: <p>I’m trying to quantize a simple model using PT2E quantization workflow and reproduced the code from the Quick Start Guide. However, the sizes of the original floating point model and the quantized model are the same. What am I missing here?</p>
<pre><code class="lang-auto">import os
import torch
from torch.export import export
from torchao.quantization.pt2e.quantize_pt2e import prepare_pt2e, convert_pt2e
import torchao.quantization.pt2e.quantizer.x86_inductor_quantizer as xiq
from torchao.quantization.pt2e.quantizer.x86_inductor_quantizer import (
    X86InductorQuantizer,
    get_default_x86_inductor_quantization_config,
)

class M(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.linear1 = torch.nn.Linear(1024, 1024)
        self.linear2 = torch.nn.Linear(1024, 1024)

    def forward(self, x):
        x = self.linear1(x)
        out = self.linear2(x)

        return out


float_model = M().eval()

example_inputs = (torch.randn(128, 1024),)

exported_model = export(float_model, example_inputs).module()

quantizer = X86InductorQuantizer()
quantizer.set_global(xiq.get_default_x86_inductor_quantization_config())

prepared_model = prepare_pt2e(exported_model, quantizer)
prepared_model(*example_inputs)
quantized_model = convert_pt2e(prepared_model)

torch.save(float_model.state_dict(), 'float_model.pt')
torch.save(quantized_model.state_dict(), 'quant_model.pt')
float_model_size_mb = os.path.getsize('float_model.pt') / 1024 / 1024
quant_model_size_mb = os.path.getsize('quant_model.pt') / 1024 / 1024

print(f'Float model size: {float_model_size_mb:.2f} MiB')
print(f'Quantized model size: {quant_model_size_mb:.2f} MiB')
</code></pre>
<p>The output on Google Colab is</p>
<pre><code class="lang-auto">Float model size: 8.01 MiB
Quantized model size: 8.01 MiB

</code></pre>
PyTorch + captum on Colab: Get it to work?

vuhung: <p>Hi everyone,</p>
<p>How to get the example below works on Google Colab?</p>
<p>(IMDB_TorchText_Interpret.ipynb)</p>
<p>!pip install torch</p>
<p>!pip install torchtext</p>
<p>!pip install captum</p>
<p>didn’t work.</p>
<p>I am trying to get it worked on Colab.</p>
<p>If I couldn’t, running locally will be the 2nd option.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/captum/blob/master/tutorials/IMDB_TorchText_Interpret.ipynb">
  <header class="source">

      <a href="https://github.com/pytorch/captum/blob/master/tutorials/IMDB_TorchText_Interpret.ipynb" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/captum</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/captum/blob/master/tutorials/IMDB_TorchText_Interpret.ipynb" target="_blank" rel="noopener nofollow ugc">tutorials/IMDB_TorchText_Interpret.ipynb</a></h4>

<div class="git-blob-info">
  <a href="https://github.com/pytorch/captum/blob/master/tutorials/IMDB_TorchText_Interpret.ipynb" rel="noopener nofollow ugc"><code>master</code></a>
</div>


      <pre><code class="lang-ipynb">{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Interpreting text models:  IMDB sentiment analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook loads a pretrained CNN model for sentiment analysis on an IMDB dataset. It makes predictions on test samples and interprets those predictions using the Integrated Gradients method.\n",
    "\n",
    "The model was trained using an open source sentiment analysis tutorials described in: https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/4%20-%20Convolutional%20Sentiment%20Analysis.ipynb with the following changes:\n",
    "\n",
    "- TEXT: set lower=True at initialization and call build_vocab() on the entire training data including validation, to avoid mismatched indices\n",
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/captum/blob/master/tutorials/IMDB_TorchText_Interpret.ipynb" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>What’s the error you are seeing?</p>

vuhung: <p>Thanks for the rep.</p>
<p>You can see the errors by loading the notebook into Colab.</p>
<p>The current code doesn’t work with Colab, of course by looking at <code>pip install</code> for required libraries.</p>
Guidance from community Members

utkarsh820: <p>Greetings to all members,</p>
<p>I recently started learning PyTorch.</p>
<p>My goal is to learn computer vision, including GenAI trends.</p>
<p>Well, the question looks like a noob one. Sorry in advance, but it can’t be helped.</p>
<p>Hoping to get quality answers that may help in my journey…</p>
<p>Right Now i’m exploring CNNs → GANs → Diffusion Models<br>
All I know is that classification, image segmentation, object tracking, and image synthesis are some parts of computer vision.</p>

twoudsma: <p>Welcome <a class="mention" href="/u/utkarsh820">@utkarsh820</a>!</p>
<p>There are many wonderful things you can build with PyTorch for CV applications. You can take a look at the image/video tutorials right here: <a href="https://docs.pytorch.org/tutorials/domains.html" class="inline-onebox" rel="noopener nofollow ugc">Domains — PyTorch Tutorials 2.8.0+cu128 documentation</a>.</p>
Enhanced PyTorch Tutorials: From Basics to Deployment

vuhung: <p>Hey PyTorch community!</p>
<p>I’ve created a comprehensive PyTorch tutorial series that takes learners to learn PyTorch from scratch. These tutorials are specifically designed for those transitioning from TensorFlow to PyTorch, like me.</p>
<p>Why I made these tutorials:</p>
<p>- I’ve struggled with the transition from TensorFlow to PyTorch</p>
<p>- I’ve found that the existing tutorials are not enough to help me learn PyTorch</p>
<p>You need a little bit of basic knowledge of Math, Python, NumPy, (or TensorFlow) to learn PyTorch.</p>
<p>These tutorials are perfect for:</p>
<p>- TensorFlow users transitioning to PyTorch</p>
<p>- Anyone wanting comprehensive PyTorch mastery</p>
<p>The content of the tutorials are:</p>
<p>- PyTorch Introduction</p>
<p>- Tensor Fundamentals (like NumPy)</p>
<p>- Autograd Explained</p>
<p>- Building Models (like LeNet-5)</p>
<p>- TensorBoard Integration (un on Colab too!)</p>
<p>- Training Models</p>
<p>- Model Interpretability (Captum)</p>
<p>Quick start:</p>
<p>Open the notebooks in Google Colab.</p>
<p>If you want to run the notebook locally, you can run the notebook in your local environment.</p>
<p>Feedback:</p>
<p>- If you have any feedback/questions, please let me know.</p>
<p>The tutorial: <a href="https://github.com/vuhung16au/pytorch-mastery/tree/main/examples/pytorch-tutorials" class="inline-onebox" rel="noopener nofollow ugc">pytorch-mastery/examples/pytorch-tutorials at main · vuhung16au/pytorch-mastery · GitHub</a></p>
Metric to save best checkpoint for diffusion model

Introvert_Loner: <p>I am trying to train a diffusion model on a specialized dataset (different from ordinary images), and what concerns me that when comparing/evaluating models in papers (e.g. the original Stable Diffusion paper), they use FID, but in the training code (the validation_step if used lightning), they use the MSE between the prediction of noise and true noise in the validation set to save the weights &amp; biases (checkpoint).</p>
<p>I know that it is computationally expensive to use FID during training as we have to do inference on a large number of samples, but does saving checkpoints according to MSE between noise align with FID? If not, we’re saving the wrong weights &amp; biases? If yes, why don’t we just use the MSE between the noise in comparing/evaluating models in papers instead of FID?</p>
How to apply selective activation checkpointing on _grouped_mm

go_brrr: <p>From the selective activation checkpointing blog, I understood that we can save compute intensive operations while recompute operations that are memory intensive. How do I add <code>_grouped_mm</code> to <code>ops_to_save</code> below?</p>
<p>Does directly adding <code>torch.ops.aten.grouped_mm.default</code> works or can I directly adding `torch._grouped_mm` does the trick ?</p>
<blockquote>
<p>ops_to_save = [</p>
<p>torch.ops.aten.mm.default,</p>
<p>torch.ops.aten.bmm.default,</p>
<p>torch.ops.aten.addmm.default,</p>
<p>torch.ops.aten.grouped_mm.default,</p>
<p><em># Save collectives that produce dynamic-shaped outputs</em></p>
<p>torch.ops._c10d_functional.all_to_all_single.default,</p>
<p>torch.ops._c10d_functional.reduce_scatter_tensor.default,</p>
<p><em># for low precision training, it’s useful to always save</em></p>
<p><em># the result of max, since the absolute maximum is</em></p>
<p><em># used to compute the scaling factor for quantization.</em></p>
<p>torch.ops.aten.max.default,</p>
<p>]</p>
<p>def _apply_ac(<em>model</em>: torch.nn.Module):</p>
<p><em>from</em> torch.utils.checkpoint <em>import</em> CheckpointPolicy, create_selective_checkpoint_contexts</p>
<p><em>from</em> torch.distributed.algorithms._checkpoint.checkpoint_wrapper <em>import</em> checkpoint_wrapper, CheckpointImpl</p>
<p><em># Create a policy function that returns a CheckpointPolicy</em></p>
<p>def policy_fn(<em>ctx</em>, <em>op</em>, *<em>args</em>, **<em>kwargs</em>):</p>
<p><em>if</em> <em>op</em> in ops_to_save:</p>
<p><em>return</em> CheckpointPolicy.MUST_SAVE</p>
<p><em>else</em>:</p>
<p><em>return</em> CheckpointPolicy.PREFER_RECOMPUTE</p>
<p>“”“Apply activation checkpointing to the model.”“”</p>
<p><em>for</em> layer_id, transformer_block <em>in</em> <em>model</em>.layers.named_children():</p>
<p>transformer_block = checkpoint_wrapper(</p>
<p>transformer_block,</p>
<p><em>checkpoint_impl</em>=CheckpointImpl.NO_REENTRANT,</p>
<p><em>context_fn</em>=partial(create_selective_checkpoint_contexts, policy_fn),</p>
<p>)</p>
<p><em>model</em>.layers.register_module(layer_id, transformer_block)</p>
<p>logger.info(“Applied selective activation checkpointing to the model”)</p>
</blockquote>
<p>I am using moe implementation from torch titan from reference, I wan</p>
PyTorch Compatibility with Older CUDA Versions

Harishankar14: <p>Hey Guys,<br>
I have been working on an ML project, so i have a GPU server (An ancient one) which is backed by CUDA 3.0 .So what is the minimum version supported for PyTorch ?</p>
<p>Thank You <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Which GPU are you using?</p>
Updating tensors that are used in backpropagation but are not network parameters

jguzik: <p>I am currently trying to reimplement “Deep SVDD” for my own project.</p>
<p>During training, I use my network to compute the outputs as usual. To compute the loss, I then use the hypersphere center c and radius R as follows:</p>
<pre><code class="lang-auto">outputs = model(data)
dist = torch.sum((outputs - self.c) ** 2, dim=1)
scores = dist - self.R ** 2
loss = self.R ** 2 + (1 / self.nu) * torch.mean(torch.max(torch.zeros_like(scores), scores))
loss.backward()
optimizer.step()

</code></pre>
<p>After backpropagating the loss and updating the parameters, I now want to update the radius R. In the original implementation this is done by</p>
<pre data-code-wrap="python"><code class="lang-python">self.R.data = torch.tensor(get_radius(dist, self.nu), device=self.device)
</code></pre>
<p>where the get_radius function is defined as follows:</p>
<pre data-code-wrap="python"><code class="lang-python">def get_radius(dist: torch.Tensor, nu: float):
    return np.quantile(np.sqrt(dist.clone().data.cpu().numpy()), 1 - nu)
</code></pre>
<p>After updating the radius, the processing of the current training batch is done.</p>
<p>Now I am wondering whether this is the proper way to do this. I know that the use of .data can be problematic and can cause issues during backpropagation. I have been reading around about it but I can’t quite wrap my head around it and how to fix it. Would it cause issues in the original implementation and how should I implement it instead to avoid any issues and why?</p>
<p>More generally, my question is how to properly update the radius in my implementation without causing issues with backpropagation or more generally how to update tensors like the radius?</p>

valerian.rey: <p>You may only have issues if you update in-place a tensor that is referenced in the graph, between the forward pass and the backward pass. This is because a reference to this tensor may be saved in the computation graph, so changing the value stored in the tensor will also change the result of the backpropagation. Since you’re updating <code>self.R</code> after the graph has been used (after <code>loss.backward()</code>), what you’re doing should never cause issues. However, it’s cleaner to just assign to <code>self.R</code> directly rather than <code>self.R.data</code> (this way you avoid making an in-place update altogether).</p>
<p>By the way, since your radius should not require grad, you could compute it in a <code>torch.no_grad()</code> context just to be sure that no operation constructing a graph will be executed (which would waste time), and that the obtained tensor does not require grad (which would make the backpropagation waste time at the next iteration).</p>
<p>So your current implementation should be fine, but a maybe safer approach would be:</p>
<pre><code class="lang-auto">with torch.no_grad():
    self.R = torch.tensor(get_radius(dist, self.nu), device=self.device)
</code></pre>

jguzik: <p>Hi, thank you a lot. That really cleared up my confusion!</p>
DistributedDataParallel init hangs

cheeks: <p>The following code hangs on the DDP initialisation when multiple GPUs are used. This happens with torchrun or when using a multiprocessing method as described here: <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html#" class="inline-onebox" rel="noopener nofollow ugc">Writing Distributed Applications with PyTorch — PyTorch Tutorials 2.5.0+cu124 documentation</a>. A single GPU run works fine.</p>
<p>Example code:</p>
<pre><code class="lang-auto">from torchvision.models import resnet18
from torch import nn
import torch
import os

rank = int(os.environ['LOCAL_RANK'])
world_size = int(os.environ['WORLD_SIZE'])
print(f"init process group {rank}")
torch.cuda.set_device(rank)
torch.distributed.init_process_group(backend="nccl", world_size=world_size, rank=rank)
device = torch.device(f"cuda:{rank}")
# torch.distributed.barrier()
model = resnet18().to(device)
print(f"starting {rank}")

new_model = nn.parallel.DistributedDataParallel(model, device_ids=[device])
print(f"done{rank}")
</code></pre>
<p>I am running it with this command:</p>
<pre><code class="lang-auto">torchrun --nnodes=1 --nproc_per_node=gpu simple.py
</code></pre>
<p>And i get this output before it hangs. Eventually it times out. A single gpu run takes about 5 seconds, so it’s not an issue with the size of the model.</p>
<pre data-code-wrap="init"><code class="lang-init">init process group 1
init process group 0
init process group 3
starting 2
starting 3
starting 1
starting 0
</code></pre>
<p>Adding --rdzv-backend=c10d --rdzv-endpoint=localhost:0 options makes no difference. As there is no error, i don’t know how to debug.<br>
Env:<br>
cuda 12.2,  Driver Version: 565.57.01, 4 x A6000 GPU. Torch 2.1.1+cu121</p>
<p>I have had success running on another machine with different GPU/driver version but same CUDA/torch version.</p>

cheeks: <p>Issue turned put to be with the NVLink setup on the GPUs i was using, so i had to disable NVLink and use PCI instead, using the exports below:</p>
<p>export NCCL_P2P_DISABLE=1 export NCCL_IB_DISABLE=1 export NCCL_SOCKET_IFNAME=lo</p>
Ticket Giveaway for PyTorch Conference - Dendritic Optimization

PerforatedAI: <p>Hello, this is Dr. Rorry Brenner, the founder of <a href="https://www.perforatedai.com" rel="noopener nofollow ugc">Perforated AI</a>. We’re one of the sponsors for the upcoming <a href="https://events.linuxfoundation.org/pytorch-conference" rel="noopener nofollow ugc">PyTorch conference</a>. As a startup sponsor they gave us 4 tickets but we’ll only be bringing 3 people and we’d love to give that extra ticket away! If you’d like to save $1000 for under an hour of your time read more details below.</p>
<p>We’ve just released an <a href="https://github.com/PerforatedAI/PerforatedAI" rel="noopener nofollow ugc">open source version</a> of our project to get started with dendritic optimization.  This is a new tool based on modern neuroscience that empowers ML engineers to build build smarter, smaller, and more accurate neural networks. The project is implemented in PyTorch and requires only a few lines of code to get started.  If you’d like to join the raffle, just throw those lines of code into a project you’re already working on, rerun your training, and submit a PR to our examples folder.  We’ll pick a winner on October 6th.</p>
<p>Considerations before entering:</p>
<ul>
<li>
<p>re-running training does take some time.  If your current project takes a week to train, this won’t be a good fit.  If it takes under 24 hours, that’s perfect.</p>
</li>
<li>
<p>Putting those few lines of code in the right places is significantly easier if you wrote all the code yourself.  If you are using an external library for your project it likely won’t be as easy.  We are already set up for Huggingface Transformers and PyTorch Lightning, but if you’re working with a different library this also might not be a good fit.</p>
</li>
<li>
<p>We’re very happy to support.  If you run your first experiment and don’t see improvements please reach out and we can help suggest some alternative dendritic hyperparameters.</p>
</li>
</ul>
<p>Happy Hacking!</p>
Customizing PyTorch images on different arch

Aurelien_MERCIER: <p>Hello guys,</p>
<p>I’ve been working extensively on customizing PyTorch images, particularly focusing on enabling and disabling various options like <code>USE_CUSPARSELT</code>, <code>USE_NCCL</code>, and others. Given the wide range of build flags involved (e.g., CUDA, cuDNN, TensorPipe, XNNPACK, MKLDNN, OpenMP, and others), managing these configurations dynamically across different environments (A100, H100, etc.) is becoming increasingly complex.</p>
<p>I’m looking for guidance on how to streamline this process effectively within PyTorch. Specifically, I want to ensure that I can enable/disable these flags seamlessly depending on the target environment while keeping the build process as efficient as possible.</p>
<p>Is there someone I could connect with who specializes in managing PyTorch builds and configurations? I’d like to discuss best practices for automating this process, especially when it comes to handling flags like <code>USE_CUSPARSELT</code>, where certain features are still under development. Any insights on how to manage this complexity effectively would be greatly appreciated.</p>
<p>Thanks in advance for your help!</p>

ptrblck: <p>Could you describe your actual use case and why you want to disable certain features?<br>
While it’s true that some features are experimental, it would still be irrelevant which device you are using. E.g. if you are not interested in using cuSPARSELt, you can just globally disable it.</p>

Aurelien_MERCIER: <p>Thanks for the input! To give you some context, we are juggling different versions of PyTorch, NCCL, Transformer Engine (TFE), and Flash Attention (FA) across our environment. Managing these versions requires us to carefully enable or disable features based on compatibility and performance needs. For example, we recently had to explicitly enable cuSPARSELt (recompile torch) to test and optimize sparse matrix operations on specific hardware because we had issues with the nightly build.</p>
<p>Our goal is to ensure that the right set of features is activated for each build, as we often deal with experimental flags and custom builds. This level of control helps us tailor the performance to the workloads we are running, especially on diverse devices like A100s and H100s.</p>
<p>Having a flexible way to manage these flags across different versions would be invaluable for us. Do you have any suggestions for making this process smoother?</p>
<p>Thanks again for your help!</p>

tbouvier: <p>Hey <a class="mention" href="/u/aurelien_mercier">@Aurelien_MERCIER</a>, your are looking for a solution like Spack.</p>
Understanding Profiler Output

Mole_Turner: <p>I’ve got the below profiler output. I’m fairly comfortable with what each of the columns represent, however I’m slightly confused as to why <code>aten::convolution_backward</code>appears to be the most time costly operation for both the GPU and CPU.</p>
<pre><code class="lang-auto">Name                        Self CPU     CPU total      Self CUDA      CUDA total
aten::convolution_backward  347.071ms    42.355s        105.888s       105.549s
</code></pre>
<p>I understand that self CPU is time spent directly within this operator’s code, excluding function calls, which infers that given the different between the self CPU and CPU total values, aten::convolution_backwardmakes some function calls that are executed on the CPU. I’m just a bit confused as I would have assumed the backward call would have been all on the GPU, given that’s where the model and the data is?</p>
How to increase TorchServe worker timeout?

Zach_Liu: <p>I’m trying to deploy a model (base image:763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-inference:2.6-gpu-py312) to SageMaker endpoint. In my model_fn()I’m loading some datasets into memory. However every time torchserve throws an error after about 2 minutes of the worker thread starting:</p>
<pre><code class="lang-auto">[ERROR] W-9000-model_1.0 org.pytorch.serve.wlm.WorkerThread - Number or consecutive unsuccessful inference 2 
[ERROR] W-9000-model_1.0 org.pytorch.serve.wlm.WorkerThread - Backend worker error 
org.pytorch.serve.wlm.WorkerInitializationException: Backend worker did not respond in given time
</code></pre>
<p>I have tried to add a model-config.yaml file with <code>responseTimeout: 1200</code>but that didn’t see to make a difference. Are there other ways to increase the torchserve worker timeouts?</p>
How am i supposed to code an axonal delay in my SNN?

Photon1: <p>Hi,</p>
<p>so i am trying to make work a convolusionnal SNN with axonal delay, with pytorch.</p>
<p>But i don’t find any info on internet about how to code an axonal delay.</p>
<p>Here i am trying to just make a kind of circular buffer. But i don’t want it to be like a list, because i want to add to it, a tensor like this 10, 0, h, w, the input, a tensor like this h, w, multiplyed by a another mask tensor, like this 10, 0, h, w, filled with ones sparsely distributed to make the axonal delays.</p>
<p>So i need this circular buffer to be a tensor and not a list, because i need to apply to it tensor operations.</p>
<p>Maybe i am doing wrong ? Is there a better way to do it ? I am looking for quickness. Or do you know exactly how am i supposed to make this circular buffer ? Should i use torch.roll ?</p>
<p>Thanks for your help!</p>
GPU OOM while not using all memory?

sad_robot: <p>I got this GPU OOM error from pytorch which seems to be self-contradictory and I don’t understand how it’s possible:</p>
<p>”””[rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 108.00 MiB. GPU 0 has a total capacity of 94.50 GiB of which 2.67 GiB is free. Including non-PyTorch memory, this process has 85.88 GiB memory in use. Of the allocated memory 77.92 GiB is allocated by PyTorch, and 6.52 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  ( <a href="https://pytorch.org/docs/stable/notes/cuda.html#environment-variables" class="inline-onebox" rel="noopener nofollow ugc">CUDA semantics — PyTorch 2.8 documentation</a> )”””</p>
<p>As the error says the GPU has 94.5GB total, yet the error also says “this process has 85.88GB memory in use” (which includes allocated and reserved). I have exclusive access to this GPU (I’m using SLURM) and there should be only 1 pytorch process. I guess one weird thing is that I’m using FSDP strategy despite having one process, just because I didn’t see a point in changing the strategy when I’m using just one GPU.</p>
<p>Anyways what happened to the missing 10GB?</p>

ptrblck: <p>You could be running into the OOM due to memory fragmentation. Did you try to use expandable segments as the error message also explains?</p>

sad_robot: <p>Thanks for your reply.</p>
<p>I didn’t try that yet, it also says: “GPU 0 has a total capacity of 94.50 GiB <strong>of which 2.67 GiB is free</strong>“. I think your right that memory fragmentation explains why it OOMed on 100MB when there is 2.67GB free, <em>but</em> I’m not sure if it explains why 94.5GB - 85.88GB = 2.67GB?</p>

ptrblck: <p>Yes, that’s a good point. Did you try to allocate the entire memory as a smoke test? Also, did you check what <code>nvidia-smi</code> was reporting?</p>
Are there any related works on "megakernel"?

X_Wang: <p>Hi all, are there any related works on “megakernel” here?<br>
I’m planning to research this topic using torch.compile.<br>
According to the article: “We bypass a key issue by merging the entire Llama-1B forward pass into a single ‘megakernel,’ eliminating kernel boundaries. On an H100, this achieves 78% memory bandwidth utilization and outperforms existing systems by over 1.5x.”</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://hazyresearch.stanford.edu/blog/2025-05-27-no-bubbles">
  <header class="source">
      <img src="https://hazyresearch.stanford.edu/favicon.ico" class="site-icon" alt="" width="48" height="48">

      <a href="https://hazyresearch.stanford.edu/blog/2025-05-27-no-bubbles" target="_blank" rel="noopener nofollow ugc">hazyresearch.stanford.edu</a>
  </header>

  <article class="onebox-body">
    <img width="460" height="460" src="https://hazyresearch.stanford.edu/hazy-logo.png" class="thumbnail onebox-avatar" alt="">

<h3><a href="https://hazyresearch.stanford.edu/blog/2025-05-27-no-bubbles" target="_blank" rel="noopener nofollow ugc">Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B</a></h3>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>