akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Impact of PCIe lane configuration on multi GPU training and inference

username: adrew25
cooked: <p>Hi new to the forum and i am a bit confused with the HW right now in the market.<br>
I’m planning a new workstation build with a Ryzen 9 9950X3D, 96GB RAM, and dual GPUs. The issue is that most X870E motherboards only provide one PCIe 5.0 x16 slot, while the other GPU would have to run on a PCIe 4.0/5.0 x4 slot (through the chipset).<br>
My main workloads are:<br>
1.Training and fine-tuning LLMs (PyTorch, DDP/HF Accelerate).<br>
2. Inference for LLMs.<br>
3. Computer vision training and inference.<br>
4! And of course causal pc usage( Playing a game etc.)<br>
My questions are:<br>
1.How much of an impact does having one GPU on PCIe x16 and the other on PCIe x4 have on training (gradient synchronization, data loading, etc.)?<br>
2. Is inference also affected significantly by the reduced bandwidth, or is it mostly negligible?<br>
3. Would it make sense to prioritize an AM5 board with x8/x8 support, or is the bottleneck only noticeable for very large models?<br>
Thanks in advance for any insights i am struggling to figure out before i give a tone of money. ( Miss the days where the hw was simpler… now the consumer hw is bloated with wifi and rgb….)</p>

username: tonyr
cooked: <p>I’m also speccing a dual (5090) box right now and was wondering the same.  At the moment I’m doing LLM research, so just passing in a batch of token IDs, very low bandwidth.  I have a local hardware builder (<a href="https://woc.co.uk/" rel="noopener nofollow ugc">https://woc.co.uk/</a>) whom I really trust as I’ve been buying computers for machine learning from them for the last 25 years.  They have convinced me to get a Xeon W5-3525 CPU with ASUS PRO WS W790E-SAGE motherboard because nothing less is going to support two 16x PCIe lanes.  The dual 5090 setup is common for them, they supply to Cambridge University and local startups, they make a business out of getting ti right.  Now, I’m no expert, but I do know that the 5090s will cost me £4,000 here in the UK, the supporting hardware is £4,200, so that’s as much for the computer as the GPUs.  This doesn’t seem unreasonable to me, it’s easy to swap out GPUs in a few years when a 6090 or 7090 comes out, it’s hard to upgrade a motherboard (I’ve done it once - never again - too many things to unplug and forget where they go).  Your game plan will be different, but I struggled with the single x16 slot issue for a while and then just gave up and went on trusting my supplier.</p>
C++/CUDA extension: build object files first, then shared lib extension

username: codeguru
cooked: <p>I have two sets of implementations in C++CUDA. Both implementations require include files with clashing definitions.</p>
<p>In CMake, I can do the following (please ignore minor mistake – making a case):</p>
<pre data-code-wrap="makefile"><code class="lang-makefile"># first object
add_library(FirstObject OBJECT src/file1.cpp src/file2.cu)
target_include_directories(FirstObject PRIVATE include_dir1)
target_set_properties(FirstObject PROPERTY POSITION_INDEPENDENT_CODE ON)
target_compile_options(FirstObject PRIVATE custom_options1)

# second object
add_library(SecondObject OBJECT src/file3.cpp src/file4.cu)
target_include_directories(SecondObject PRIVATE include_dir2)
target_set_properties(SecondObject PROPERTY POSITION_INDEPENDENT_CODE ON)
target_compile_options(SecondObject PRIVATE custom_options2)

# final shared lib
add_library(FinalSharedLib SHARED $&lt;TARGET_OBJECTS:FirstObject&gt; $&lt;TARGET_OBJECTS:SecondObject&gt; shared_library.cpp)
...
</code></pre>
<p>I want to do the same in Setup tools. The best solution I came across was writing custom build class, based on <code>build_ext</code>, but could not extend it to use CudaExtension.</p>
<p>An existing example will be helpful.</p>

username: codeguru
cooked: <p>I solved this by subclassing BuildExtension. This works very well for C++ files. For CUDA files, I simply invoke NVCC compile commands via Python Subprocess lib.</p>
<p>Better solution would be to set compiler to NVCC and then use the same interface as C++ files. But this would need additional settings.</p>
Installation Candidates

username: mutineering
cooked: <p>Hello..   I am on ubuntu noble 24.04 with python 3.12 on a desktop machine which has a working installation of CUDA13 and a couple of graphics cards.  I can’t use pip to install torch without the ‘externally-managed-environment’ error…..   and sudo python3-pytorch gives the ‘no installation candidate’ error.  Am I doing something wrong?  Or do I just have to wait until an installation candidate appears…  or is there something else I can safely do in the meantime?</p>

username: ptrblck
cooked: <p>Install a virtual Python environment and the first error should be gone.</p>

username: mutineering
cooked: <p>All good and up and running. Thanks.</p>
<pre><code class="lang-auto">import torch
&gt;&gt;&gt; torch.cuda.is_available()
True
</code></pre>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=14" title=":slightly_smiling_face:" class="emoji only-emoji" alt=":slightly_smiling_face:" loading="lazy" width="20" height="20"></p>
Is UVM supported?

username: johnnynunez
cooked: <aside class="onebox allowlistedgeneric" data-onebox-src="https://developer.nvidia.com/blog/whats-new-in-cuda-toolkit-13-0-for-jetson-thor-unified-arm-ecosystem-and-more/">
  <header class="source">
      <img src="https://developer-blogs.nvidia.com/wp-content/themes/nvidia/dist/images/favicon_300a1064.ico" class="site-icon" alt="" width="48" height="48">

      <a href="https://developer.nvidia.com/blog/whats-new-in-cuda-toolkit-13-0-for-jetson-thor-unified-arm-ecosystem-and-more/" target="_blank" rel="noopener nofollow ugc" title="04:00PM - 02 September 2025">NVIDIA Technical Blog – 2 Sep 25</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:600/338;"><img src="https://developer-blogs.nvidia.com/wp-content/uploads/2025/08/hpc-social-jetson-jet-pack-1200x675-1.png" class="thumbnail" alt="" width="690" height="388"></div>

<h3><a href="https://developer.nvidia.com/blog/whats-new-in-cuda-toolkit-13-0-for-jetson-thor-unified-arm-ecosystem-and-more/" target="_blank" rel="noopener nofollow ugc">What’s New in CUDA Toolkit 13.0 for Jetson Thor: Unified Arm Ecosystem and...</a></h3>

  <p>The world of embedded and edge computing is about to get faster, more efficient, and more versatile with the upcoming CUDA 13.0 release for Jetson Thor SoC powered by NVIDIA Blackwell GPU architecture.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: marksaroufim
cooked: <p>Not quite, we have talked a lot about it though and the main issue is UVM has different semantics per vendor and it’s not quite clear what PyTorch should be doing</p>
<p>There’s a bunch of interesting history <a href="https://github.com/pytorch/pytorch/issues/44380" class="inline-onebox">Support for NVIDIA UVM technology · Issue #44380 · pytorch/pytorch · GitHub</a></p>

username: ptrblck
cooked: <p>The <code>MemPool</code> API should allow you using managed memory in case that’s the use case. However, since Jetson devices are using physically unified memory by design I’m unsure what the use case is since no information was given.</p>

username: johnnynunez
cooked: <p>I was playing with GH200, with Qwen3-Next. But it was surprisingly to me that only take 96GB for H100. When GH200(480GB) is available</p>

username: ptrblck
cooked: <p>You were sharing a link to Jetson Thor so it’s also unclear to me how it’s related to your GH system. In any case, check the <code>MemPool</code> docs to see how UVM can be used.</p>
[Question]CUDA error: no kernel image is available for execution on the device

username: fiona-fifi
cooked: <p>Hi, I am trying to install and use PyTorch on my machine, but something went wrong:</p>
<pre><code class="lang-auto">AcceleratorError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
</code></pre>
<p>My system specifications are given below:</p>
<pre><code class="lang-auto">OS: Windows 10
Cuda: Cuda compilation tools, release 12.6, V12.6.85
Device name: NVIDIA GeForce RTX 5090 D
torch version: 2.8.0+cu126
python version: 3.1.1
</code></pre>
<p>Can someone help me solve this problem? Thank you so much!</p>

username: ptrblck
cooked: <p>Uninstall the current PyTorch binary and install any stable or nightly PyTorch binary built with CUDA 12.8+ which is supporting your Blackwell GPU.</p>
Why torch.profile give slower exec time then time.prof_counter with torch.cuda.current_stream(self.device).synchronize()?

username: HuangChiEn
cooked: <p>As title described, the following code snippet give different results (not minor difference) ?</p>
<p>torch profile : CPU time total : 446.860ms, CUDA time total : 122.922ms</p>
<p>time.perf_counter : 0.3481s ~ 348.1ms (total elapsed time of <code>self.model(model_input)</code>)</p>
<p>Code snippet i used to profile :</p>
<pre><code class="lang-auto"># torch profile
self.model.eval()

for _ in range(30):
    _, _, _ = self.model(model_input)
print('warm up done!')
            
with torch.no_grad():
     torch.cuda.current_stream(self.device).synchronize()
     with profile(activities=[ProfilerActivity.CUDA, ProfilerActivity.CPU], with_stack=True, with_flops=True) as prof:
          pred_speed_wps, pred_route, language = self.model(model_input)
          prof.step()

      print(prof.key_averages(group_by_stack_n=5).table(sort_by="flops"))
</code></pre>
<p>vs.</p>
<pre><code class="lang-auto"># time.perf_counter

class Tch_prof(object):
    def __init__(self, device):
        self.device = device
        self.hw_type = 'gpu'
        self.tlt_time = {
            'cpu' : 0,
            'gpu' : 0
        }

    def __enter__(self):
        torch.cuda.current_stream(self.device).synchronize()
        self.s = time.perf_counter()
         
    def __exit__(self, *exc):
        torch.cuda.current_stream(self.device).synchronize()
        self.tlt_time[self.hw_type] += time.perf_counter() - self.s
        
    def get_profile(self, hw_type='all'):
        if hw_type == 'all':
            return self.tlt_time
        elif hw_type in self.tlt_time.keys():
            return self.tlt_time[hw_type]
        else:
            raise RuntimeError(f"No such hardware type {hw_type}") 

# run_step code snippet :
self.model.eval()

for _ in range(30):
    _, _, _ = self.model(model_input)
    print('warm up done!')
            
    with torch.no_grad():
         prof = Tch_prof(device=self.device)
         with prof:
              pred_speed_wps, pred_route, language = self.model(model_input)
        
    print(prof.get_profile())
</code></pre>
<p>For using time.prof_counter, i already had applied torch.cuda.current_stream(self.device).synchronize() and expect this is the correct way to measure the time elapsed, but still different from the torch profile…orz ?</p>
<p>If so, which one is the correct way to do time benchmarking of torch model in this case ?</p>
<p>Any suggestion will be helpful ~</p>
what  happens when I use torch.profiler.profile with activities=[torch.profiler.ProfilerActivity.CPU, torch.profiler.ProfilerActivity.CUDA]

username: qq1243196045
cooked: <p>I use profiler with acitvities=[torch.profiler.ProfilerActivity.CPU, torch.profiler.ProfilerActivity.CUDA] ,and get time 0.22ms in cpu time avg,5.14us in cuda time avg, and when I use time.time() with torch.cuda.synchronize(),the result is 0.24 ms,What is the difference between these results？<br>
my code is like<br>
`activities = [torch.profiler.ProfilerActivity.CPU, torch.profiler.ProfilerActivity.CUDA]<br>
torch.cuda.synchronize()<br>
start = time.time()<br>
with torch.profiler.profile(activities= activities, record_shapes=True, profile_memory=False) as pf:<br>
output = model(input)<br>
torch.cuda.synchronize()<br>
end = time.time()<br>
runtime = end-start()</p>
<p>`</p>

username: HuangChiEn
cooked: <p>0.24ms is quite close with 0.22ms+5.14us ; it seems give similar results between torch.profile &amp; time.perf_counter + sync op.</p>
<p>i can’t see where’s the issue ?</p>
This error message always shows up

username: gaopeifeng
cooked: <p>Continuing the discussion from <a href="https://discuss.pytorch.org/t/exception-raised-from-recvbytes-at-pytorch-torch-csrc-distributed-c10d-utils-hpp-675-most-recent-call-first/223117">Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:675 (most recent call first):</a>:</p>
<aside class="quote no-group quote-modified" data-username="gaopeifeng" data-post="1" data-topic="223117">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/b9bd4f/48.png" class="avatar"><a href="https://discuss.pytorch.org/t/exception-raised-from-recvbytes-at-pytorch-torch-csrc-distributed-c10d-utils-hpp-675-most-recent-call-first/223117/1">Exception raised from recvBytes at /pytorch/torch/csrc/distributed/c10d/Utils.hpp:675 (most recent call first):</a></div>
<blockquote>
<p>Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):</p>
<p>This error message always shows up.</p>
</blockquote>
</aside>

username: ptrblck
cooked: <p>Could you post clarifying posts into the original topic instead of creating new topics, please?</p>
Understanding relation of FSDP and TP

username: vadimkantorov
cooked: <p>I’m trying to understand to understand the example in <a href="https://docs.pytorch.org/tutorials/recipes/distributed_device_mesh.html#how-to-use-devicemesh-for-your-custom-parallel-solutions" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with DeviceMesh — PyTorch Tutorials 2.8.0+cu128 documentation</a></p>
<pre><code class="lang-auto">from torch.distributed.device_mesh import init_device_mesh
mesh_3d = init_device_mesh("cuda", (2, 2, 2), mesh_dim_names=("replicate", "shard", "tp"))

# Users can slice child meshes from the parent mesh.
hsdp_mesh = mesh_3d["replicate", "shard"]
tp_mesh = mesh_3d["tp"]
</code></pre>
<p>Doesn’t FSDP already shard the params/optimizer states/gradients across the nodes? What does it mean to do TP, when already FSDP-style sharding is applied?</p>
<p>And another question for <a href="https://docs.pytorch.org/tutorials/intermediate/TP_tutorial.html#combine-tensor-parallel-with-fully-sharded-data-parallel-together" class="inline-onebox" rel="noopener nofollow ugc">Large Scale Transformer model training with Tensor Parallel (TP) — PyTorch Tutorials 2.8.0+cu128 documentation</a> which suggests using FSDP across nodes and TP within the node. Doesn’t using FSDP across nodes not also auto-shard params/optimizer states/gradients? Or does this tutorial suggests configuring <code>fully_shard</code> without sharding params/optimizer states/gradients to only keep the model replication aspect? In other words, doesn’t FSDP always does some sort of (inefficient) tensor parallelism in the form of sharding params across ranks? And how does TP disable the FSDP’s default behavior of “gather all shards on every GPU before layer forward“? Or how does using TP change the default’s FSDP sharding of params between the GPUs? And how does using TP change the default’s FSDP communication patterns at every layer forward?</p>
<p>Often (for simplicity? for brevity?) these tutorials are missing the bits which are actually harder to understand…</p>
<p>Thanks!</p>
Support for Ulysses/Ring distributed attention for long-context training (32k) for 32B dense models

username: vadimkantorov
cooked: <p>Hi <a class="mention" href="/u/xwu">@XWu</a> !</p>
<p>I’m reading <a href="https://arxiv.org/abs/2405.07719v3" rel="noopener nofollow ugc">https://arxiv.org/abs/2405.07719</a> which proposed a unified distributed attention, building up on both Ulysses and Ring attention. Does PyTorch support something like this out-of-the-box? or as part of torchtune/torchtitan?</p>
<p>I’m trying to make my head around TP/SP versus Ulysses/Ring: whether these combine and what could be a reasonable baseline set-up for single-box 72B+32k model fine-tuning</p>
<p>How does <code>torch.distributed.tensor.experimental.context_parallel</code> compare to Ulysses? Verl framework relies on Ulysses, so trying to understand the trade-offs</p>
<p>If <code>torch.distributed.tensor.experimental.context_parallel()</code> just modified SDPA function, why not also extend the SDPA function and add in an explicit way of passing extra options to a selected backend (e.g. could be added a new backend <code>ring</code>)? Or even introduce a separate function like <code>F.ring_attention</code> (similar how <code>flex_attention</code> was added and for some reason not integrated to the SDPA frontend API). Modulating functional behavior with contexts often obscures what’s going on more than improves “works without modifications“…</p>
<p>Also, does TP and <code>torch.distributed.tensor.experimental.context_parallel</code> reuse <code>DTensor</code> for representing sharded tensors? If so, why seprate API <code>context_parallel_unshard</code> for unsharding?</p>
<p>It is also extremely confusing that <code>torch.distributed.tensor.experimental._attention.set_rotate_method</code> is global and not a method on the CP context object…</p>
<p>Thanks!</p>
Permute inside nn.Sequential

username: nada
cooked: <p>I have a convolutional layer defined inside a sequential model and would like to permute its output. Does PyTorch have an equivalent to ´´x = x.permute(0, 2, 1)´´ that can be used inside ´´nn.Sequential´´?</p>

username: ptrblck
cooked: <p>I’m not aware of a built-in module, but you can easily create your own <code>Permute</code> layer:</p>
<pre data-code-wrap="python"><code class="lang-python">class Permute(nn.Module):
    def __init__(self, dims):
        super().__init__()
        self.dims = dims
        
    def forward(self, x):
        x = x.permute(self.dims)
        return x

model = nn.Sequential(
    nn.Linear(10, 20),
    Permute(dims=[0, 2, 1]),
    nn.Linear(10, 20),
)

x = torch.randn(1, 10, 10)
out = model(x)
print(out.shape)
# torch.Size([1, 20, 20])
</code></pre>

username: Bjorn_Lindqvist
cooked: <p>See also: <a href="https://github.com/pytorch/pytorch/issues/128412" class="inline-onebox" rel="noopener nofollow ugc">A Permute Layer for torch.nn.Sequential · Issue #128412 · pytorch/pytorch · GitHub</a> And press thumbs up. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
In resolving DispatchKeySet, is PythonDispatch called first or last?

username: jinsun-yoo
cooked: <p>I posted a series of questions on the forum today. I’ve listed them in hope that putting them together would better shed light on what I do know and don’t know.</p>
<ul>
<li><a href="https://discuss.pytorch.org/t/in-dynamo-aotautograd-why-run-faketensor-through-the-code-multiple-times/223093">In Dynamo+AOTAutograd, why run Faketensor through the code multiple times?</a></li>
<li>In resolving DispatchKeySet, is PythonDispatch called first or last? (this post)</li>
<li><a href="https://discuss.pytorch.org/t/better-understanding-why-aotautograd-decomposes-fused-rms-norm-backward-for-cuda-but-not-for-meta-tensors/223096">Better understanding why AOTAutograd decomposes <code>fused_rms_norm_backward</code> for CUDA, but not for Meta tensors</a></li>
</ul>
<p>When introducing <code>__torch_dispatch__</code> or <code>TorchDispatchMode</code>, some posts seem to suggest that the PythonDispatch is being called after features such as Autograd, etc. right before the actual function is being called. For example, <a href="https://docs.pytorch.org/docs/main/notes/extending.html#torch-dispatch-calling-convention" rel="noopener nofollow ugc">this dev note</a> seems to suggest that <code>Python</code> dispatcher comes <em>after</em> <code>Autograd</code>, <code>Autocast</code>, etc.</p>
<p>However, if I look at <code>DispatchKey.h</code>, the <code>PythonDispatch</code> key is set at enum/bit # 63. The implementation looks at the most significant bit, which makes me think <code>PythonDispatch</code> has the highest priority.</p>
<p>Am I misunderstanding something here, or is the dev note ‘outdated’?</p>
An error regarding the 5070TI not supporting SM_120

username: qiang
cooked: <p>NVIDIA GeForce RTX 5070 Ti with CUDA capability sm_120 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_61 sm_70 sm_75 sm_80 sm_86 sm_90 compute_37.<br>
If you want to use the NVIDIA GeForce RTX 5070 Ti GPU with PyTorch, please check the instructions at <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>
<p>C:\Users\hq2&gt;nvcc --version<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2025 NVIDIA Corporation<br>
Built on Wed_Jan_15_19:38:46_Pacific_Standard_Time_2025<br>
Cuda compilation tools, release 12.8, V12.8.61<br>
Build cuda_12.8.r12.8/compiler.35404655_0</p>
<p>C:\Users\hq2&gt;python --version<br>
Python 3.13.2</p>
<p>C:\Users\hq2&gt;pip --version<br>
pip 25.0.1 from C:\Users\hq2\AppData\Local\Programs\Python\Python313\Lib\site-packages\pip (python 3.13)</p>
<p>C:\Users\hq2&gt;python<br>
Python 3.13.2 (tags/v3.13.2:4f8bb39, Feb  4 2025, 15:23:48) [MSC v.1942 64 bit (AMD64)] on win32<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
print(torch.<strong>version</strong>)<br>
2.8.0.dev20250317+cu128</p>
</blockquote>
</blockquote>
</blockquote>
<p>I need help</p>

username: ptrblck
cooked: <p>The nightly binary with CUDA 12.8 supports Blackwell so make sure you have a single PyTorch installation in your environment.</p>

username: frank85729
cooked: <p>help me  ..  I am rtx 5070. but this errorTraceback (most recent call last):<br>
File “c:\Users\Lenovo\Desktop\Comprehensive_floder\llm-project\test_rtx5070.py”, line 1, in<br>
import torch, time, platform, sys<br>
File “D:\app\Lib\site-packages\torch_<em>init</em>_.py”, line 281, in<br>
<em>load_dll_libraries()<br>
File "D:\app\Lib\site-packages\torch_init</em>.py", line 264, in _load_dll_libraries<br>
raise err<br>
OSError: [WinError 1114] 动态链接库(DLL)初始化例程失败。 Error loading “D:\app\Lib\site-packages\torch\lib\c10.dll” or one of its dependencies.</p>
In Dynamo+AOTAutograd, why run Faketensor through the code multiple times?

username: jinsun-yoo
cooked: <p>I posted a series of questions on the forum today. I’ve listed them in hope that putting them together would better shed light on what I do know and don’t know.</p>
<ul>
<li>In Dynamo+AOTAutograd, why run Faketensor through the code multiple times? (this post)</li>
<li><a href="https://discuss.pytorch.org/t/in-resolving-dispatchkeyset-is-pythondispatch-called-first-or-last/223094">In resolving DispatchKeySet, is PythonDispatch called first or last?</a></li>
<li><a href="https://discuss.pytorch.org/t/better-understanding-why-aotautograd-decomposes-fused-rms-norm-backward-for-cuda-but-not-for-meta-tensors/223096">Better understanding why AOTAutograd decomposes <code>fused_rms_norm_backward</code> for CUDA, but not for Meta tensors</a></li>
</ul>
<p>When tracing a FX Graph through Dynamo &amp; Lowering it to ATen IR through AOTAutograd, for the forward pass, it looks like the FakeTensor inputs are run through the whole code three times:</p>
<ol>
<li>Within Dynamo, when interpreting the Bytecode <code>CALL</code>, to get the FakeTensor output of the result of this <code>CALL</code>, to push into the stack as input to the next functions.: <code>wrap_fx_proxy</code> → … → <code>get_fake_value</code> → .. <code>run_node</code></li>
<li>Within AOTAutograd, within <code>create_aot_state</code>, to find out about mutations, etc: <code>aot_autograd.py:L575</code></li>
<li>Within AOTAutograd, within <code>aot_stage1_graph_capture</code>, to 1) Obtain the ATen IR level graph and 2) To record operations in the Autograd tape, to so that the backward pass can be traced through: <code>aot_dispatch_autograd_graph</code> → … <code>make_fx</code> → …<code>Tracer.trace</code></li>
</ol>
<p>Question: Why do we have to run FakeTensor through the function multiple times?<br>
<a href="https://dev-discuss.pytorch.org/t/torchdynamo-update-6-training-support-with-aotautograd/570" rel="noopener nofollow ugc">This DevDiscussion</a> seems to suggest that AOTAutograd is needed to reach into C++ world where the Automatic Differentiation engine sits (quote below).</p>
<p>However, even from Dynamo, we <em>do</em> go into C++ land, and the Dispatcher is able to throw things back to Python land with <code>__torch_dispatch__</code>. Couldn’t we just throw in some mode/contexts and, when Dynamo is dispatching the functions, make Autograd record the tape? (i.e. Dispatch the Autograd feature)</p>
<p>(Quote from above link)</p>
<blockquote>
<p>Training adds challenges because the PyTorch Automatic Differentiation engine sits below the PyTorch dispatcher in C++. Therefore, the operators running in the backward pass are not directly visible to TorchDynamo at the Python level.</p>
</blockquote>
WebDataset Multi-GPU Single-Node

username: ldellal
cooked: <p>Hello,</p>
<p>I’m running single-node, multi-GPU training using data in WebDataset format (~4000 shards, each with 1000 <code>.flac</code> files). My goal is to train for <code>N</code> epochs, where each epoch consists of <code>config["num_batches_per_epoch"]</code> optimizer steps.</p>
<p>Here’s what I want to achieve:</p>
<ul>
<li><strong>Continuous sampling</strong> across epochs, wrapping around when all data have been used.</li>
<li><strong>Proper reshuffling</strong> at both the shard and sample levels on every epoch.</li>
<li><strong>Different data per GPU</strong>, using <code>DistributedSampler</code>-like behavior.</li>
<li>And <strong>most importantly</strong>: ensuring that running the training long enough will eventually sample <strong>all the available data</strong> and not just a subset of them.</li>
</ul>
<p>Here’s the dataset definition I’m currently using:</p>
<pre data-code-wrap="python"><code class="lang-python">train_dataset = (
    wds.WebDataset(
        config["train_data"],
        resampled=True,
        shardshuffle=True,
        handler=wds.handlers.warn_and_continue,
        nodesplitter=wds.shardlists.split_by_node,
    )
    .shuffle(config["shuffle_buffer_size"])
    .decode(wds.torch_audio)
    .map(lambda sample: sample["flac.flac"][0])
    .with_epoch(
        config["num_batches_per_epoch"]
        * config["batch_size"]
        // config["num_workers"]
    )
)

train_dataloader = DataLoader(
    train_dataset,
    batch_size=config["batch_size"],
    num_workers=config["num_workers"],
    worker_init_fn=seed_worker,
    pin_memory=(device.type == "cuda"),
)
</code></pre>
<p>Is this the right way to do it?</p>
<p>Thanks in advance!</p>

username: H-Huang
cooked: <p>Not really familiar with <code>WebDataset</code>, how come you aren’t using <code>DistributedSampler</code> in the dataloader? Is it because wds already handles the chunking?</p>

username: ldellal
cooked: <p>I did some tests on a small dataset and manually checked how the utterances are split between processes at each epoch. This seems to be indeed the correct setup.</p>

username: ldellal
cooked: <p>Exactly, wds takes care of that</p>
DDP overwriting a buffer with random values

username: pytorcher
cooked: <p>When I call the forward function of my model it looks like buffer values are being overwritten.  I have created a DDP and in my training loop I have the following block:</p>
<pre><code class="lang-auto">print('1')
print(id(net.module.layer1.subModule1.subModules2[0].buffer1))
print(id(net.module.layer2.subModule1.subModules2[0].buffer1))
print(net.module.layer1.subModule1.subModules2[0].buffer1)
print(net.module.layer2.subModule1.subModules2[0].buffer1)
output = net(inputs)
</code></pre>
<p>then in my network definition I have:</p>
<pre><code class="lang-auto">def forward(self, x):
    print('start')
    print(id(self.layer1.subModule1.subModules2[0].buffer1))
    print(id(self.layer2.subModule1.subModules2[0].buffer1))
    print(self.layer1.subModule1.subModules2[0].buffer1)
    print(self.layer2.subModule1.subModules2[0].buffer1)
</code></pre>
<p>This prints:</p>
<blockquote>
<p>1<br>
132397489023616<br>
132397489023696<br>
tensor([1], device=‘cuda:1’)<br>
tensor([1], device=‘cuda:1’)<br>
start<br>
132397489023616<br>
132397489023696<br>
tensor([1233784], device=‘cuda:1’)<br>
tensor([0], device=‘cuda:1’)</p>
</blockquote>
<p>So clearly something about calling forward is overwriting values.  What could be causing this?  It pretty consistently happens right at this line, and its always the first batch but not always the first epoch, so I dont think its just a random race condition from the other thread.  I have seen other threads mention that overwriting is an expected condition, but the value is just set to 1 in the beginning and is never changed anywhere so these random values are not being directly copied over.</p>

username: yf225
cooked: <p>Do you have a full script for repro? We can look into it.</p>
How to choose methods to bind C++ kernel in Pytorch

username: Hi_u_Bui_Nguy_n_Trun
cooked: <p>Hi,</p>
<p>As far as i know, there two ways to bind Cpp/cuda code to Pytorch.</p>
<ol>
<li>Write cpp/cuda  code, and use <code>ctypes</code> or <code>pybind11</code> to make the normal python function, that will load and call cpp funciton. Then use <code>torch.library.register_kernel</code> to register that python function. See and <a href="https://github.com/bitsandbytes-foundation/bitsandbytes/blob/6a07ffe024e0eefd39a75f05a81602643747f071/bitsandbytes/backends/cuda/ops.py#L302" rel="noopener nofollow ugc">here</a> for the details.</li>
<li>Using <code>TORCH_LIBRARY</code> and <code>TORCH_LIBRARY_IMPL</code> macros, as describe in <a href="https://docs.pytorch.org/tutorials/advanced/cpp_custom_ops.html" rel="noopener nofollow ugc">this</a> .</li>
</ol>
<p>I want to implement my custom fused kernel in cpp/cuda code ( combine many operations in one kernel). What’s the method I should choose ?</p>
How to freeze a single neurons?

username: Junlin1111
cooked: <p>In practical development, I have tried to “freeze” a single weight element (not the entire layer or the entire parameter tensor) in a model. The purpose is to make this element not participate in the gradient calculation during backpropagation, thereby reducing part of the computational overhead. For example, for the weight tensor of an nn.Linear layer, I once tried to set it through layer.weight[0][0].requires_grad = False. However, subsequent verification found that requires_grad is an overall property of the parameter tensor in PyTorch and cannot be set for a single element alone. Even if the above assignment is executed, during backpropagation, this element will still be calculated for the gradient, and the optimizer will also update it, which does not achieve the effect of “freezing + acceleration”.</p>

username: J_Johnson
cooked: <p>Any approach to freezing single tensor elements is not going to eliminate calculation overhead. You can copy the tensor elements you want unchanged, then overwrite them after backprop with no_grad. Or you could apply a mask in a custom optimizer, like this:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim

# Simple model
model = nn.Linear(2, 2)

# Mask: 0 to freeze, 1 to train (e.g., freeze weight[0,1])
mask = torch.ones_like(model.weight)
mask[0, 1] = 0

# Custom optimizer step to apply mask
class MaskedSGD(optim.SGD):
    def step(self, closure=None):
        for group in self.param_groups:
            for p in group['params']:
                if p.grad is not None:
                    # Apply mask to grad before update (assuming mask is accessible)
                    p.grad.data.mul_(mask)
        super().step(closure)

# Use the custom optimizer
optimizer = MaskedSGD(model.parameters(), lr=0.1)

# Proceed as usual (forward, loss, backward, step)
# ... (similar to previous example)
</code></pre>
<p>On a side note, when we use the term <strong>neuron</strong>, in reference to a neural network, this is composed of a row of tensor elements within the weight tensor, and the corresponding element within the bias vector. While a singular value in the weight tensor is referred to as a <strong>tensor element</strong>. But I was still able to get your meaning of neuron from your code example.</p>
Will DataLoader deallocate the CPU memory after one batch training?

username: Zoe_Zhao
cooked: <p>Hi!</p>
<p>For my experience, when training a Dataset with Pytorch, the CPU occupied memory will keep stable. Is that because the DataLoader automatically deallocates the CPU memory occupied by the previous batch of torch tensors? May I ask how it happened? Where can I know the more details about the DataLoader?</p>

username: valerian.rey
cooked: <p>This is automatically handled by Python’s <a href="https://docs.python.org/3/library/gc.html" rel="noopener nofollow ugc">garbage collector</a>. When a Tensor stops being referenced, it will be garbage collected after a while (not necessarily right after the batch has been used) and will thus stop using memory.</p>
Compile: `DataLoader.__setattr__` should not be traced

username: Laurent1
cooked: <p>I ran into an issue with pytorch 2.7.1 with compile on CPU. I got:</p>
<blockquote>
<p>[…]</p>
<p>File “/home/laurent/train.py”, line 1358, in forward_backward_pass_model</p>
<p>configs = model(batch)</p>
<p>File “/home/laurent/python3.13_pytorch2.7_/lib/python3.13/site-packages/torch/nn/modules/module.py”, line 1751, in _wrapped_call_impl</p>
<p>return self._call_impl(*args, **kwargs)</p>
<p>~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^</p>
<p>File “/home/laurent/python3.13_pytorch2.7_/lib/python3.13/site-packages/torch/nn/modules/module.py”, line 1762, in _call_impl</p>
<p>return forward_call(*args, **kwargs)</p>
<p>File “/home/laurent/python3.13_pytorch2.7_/lib/python3.13/site-packages/torch/_dynamo/eval_frame.py”, line 659, in _fn</p>
<p>raise e.with_traceback(None) from None</p>
<p>torch._dynamo.exc.Unsupported: Attempted to inline function marked as skipped</p>
<p>Explanation: Dynamo developers have intentionally marked that the function `DataLoader._<em>setattr</em>_` should not be traced.</p>
<p>Hint: Avoid calling the function `DataLoader._<em>setattr</em>_`.</p>
<p>Hint: Remove the function `DataLoader._<em>setattr</em>_` or the file `/home/laurent/python3.13_pytorch2.7_/lib/python3.13/site-packages/torch/utils/data/dataloader.py` from torch/_dynamo/trace_rules.py. More graph breaks may occur as a result of attempting to trace into the function.</p>
<p>Hint: Please file an issue to PyTorch.</p>
<p>Developer debug context: qualname: DataLoader._<em>setattr</em>_, name: _<em>setattr</em>_, filename: `/home/laurent/python3.13_pytorch2.7_/lib/python3.13/site-packages/torch/utils/data/dataloader.py`, skip reason: skipped according trace_rules.lookup MOD_SKIPLIST</p>
</blockquote>
<p>The code is private and very large so I cannot provide the code so as to reproduce this issue.</p>
<p>It is not clear to me what I could do to solve this issue.</p>
<p>Thanks for any help.</p>

username: marksaroufim
cooked: <p>Are you compiling your data loader? That doesn’t seem right, you should check if in the forward function of the model you’re doing data loader stuff</p>
<p>Typically for issues like this train compiling the smallest unit you can before compiling an entire training loop</p>

username: Laurent1
cooked: <p>I am not compiling the data loader otherwise the error message would have made sense to me, I do</p>
<blockquote>
<p>model = torch.compile(model, backend = backend, mode = “max-autotune”, fullgraph = True)</p>
</blockquote>
<p>I will follow your advice and try compiling the smallest units first.</p>
Writing testcases for full_backward_hook

username: PyDevC
cooked: <p>I was trying to contribute to PyTorch, I tired to solve the issue <a href="https://github.com/pytorch/pytorch/issues/99556" class="inline-onebox" rel="noopener nofollow ugc">torch.func.jacrev fails if model contains full_backward_hook · Issue #99556 · pytorch/pytorch · GitHub</a></p>
<p>and made a PR: <a href="https://github.com/pytorch/pytorch/pull/161266" class="inline-onebox" rel="noopener nofollow ugc">BackwardHookFunction implementation for full_backward_hook in pytorch 2.0 format by PyDevC · Pull Request #161266 · pytorch/pytorch · GitHub</a></p>
<p>things were smooth until the maintainer asked me write testcases for the changes that I made. He did tried to explain what I have to do but I am unable to write testcases for it.</p>
<p>I would love if someone explains me what I have to test and how can I achieve it</p>
GPU MEM% allocation vs batch size and temporal dimension

username: Ryan_Xu
cooked: <p>I have a vision transformer training where the input has shape (B, T, C, H, W).</p>
<p>I notice that when I double the batch size B, GPU MEM% also increases by ~2x. On the other hand, when I compare single timestep training (T=1) with temporal trainings (e.g. T=16), it looks like in this particular example MEM% only increases by ~2x instead of 16x.</p>
<p>I tried to look at the shape of intermediate tensors and most of them have 16x more elements, as expected. Note that I use <code>flash_attn</code>, checkpointing and <code>factorized_space_time_attn</code>. I am not exactly sure how they affect the memory footprint though.</p>
<p>I would like to understand why the MEM% does not increase proportionally. Thanks for any insight!</p>

username: Arunprakash-A
cooked: <p>In transformers, the space complexity (memory) increases <strong>linearly</strong> with batch size and <strong>quadratically</strong> with sequence length or the number of tokens (not to be confused with the number of frames).</p>
<p>However, different ways of <strong>extracting tokens from frames</strong> (for example, UniformTemporalSubsample, tublets) can significantly reduce the memory requirement, since they change how many tokens are actually passed into the transformer. In addition, patch size also affects the number of tokens. All these settings affect the memory requirement.</p>

username: Ryan_Xu
cooked: <p>In my setup, the batch size and temporal channels are flattened into one channel of dimension B*T, before passing into the GlobalTranaformerLayer. So it is unclear why MEM grows linearly wrt to B while sub linearly wrt T.</p>

username: Arunprakash-A
cooked: <p>What’s the process that takes these flattened tensors and turns them into patch embeddings? And how many patch embeddings go into the transformer when T=1 and when T=16?</p>
OSError: libtorch_cuda.so: cannot open shared object file: No such file or directory

username: Mathews_Vinoy
cooked: <pre><code class="lang-auto"> 2 import torch
      3 from torch.utils.data import Dataset
----&gt; 4 import torchaudio
      5 import torchaudio.transforms as T
</code></pre>
<p>i do not have gpu</p>
<p>i installed torchaudio</p>

username: ptrblck
cooked: <p>Did you install <code>torchaudio</code> with CUDA support and PyTorch without?</p>
<p>If so, you should to install the same accelerator support, i.e. the CPU-only binaries since you do not have a GPU in your system.</p>
<p><code>pip list | grep torch</code> should show which binaries you have installed.</p>

username: Mathews_Vinoy
cooked: <p>torch                             2.8.0+cpu<br>
torchaudio                        2.8.0<br>
torchvision                       0.23.0+cpu</p>
<p>there not showing cpu at torchaudio</p>

username: ptrblck
cooked: <p>Thanks for confirming my guess.<br>
Uninstall the <code>torchaudio</code> binary with CUDA support and install the CPU-only binary:</p>
<pre><code class="lang-auto">pip uninstall torchaudio -y
pip install torchaudio --index-url https://download.pytorch.org/whl/cpu
</code></pre>

username: Mathews_Vinoy
cooked: <p>thanks it is working now</p>
Calculating Marginalloglikelihood

username: Aravind_vel
cooked: <p>Hello,</p>
<p>I’m working with <code>gpytorch</code> and am confused about the output shape of <code>ExactMarginalLogLikelihood</code>. I’m using a <code>SingleTaskGP</code> with 10 training data points, but the MLL returns 10 loss values instead of a single scalar. Could someone please help me understand why this is happening and how to get a single loss value for optimization?</p>
<p>Here is a minimal, reproducible example:</p>
<pre><code class="lang-auto">import torch
from botorch.models.transforms import Normalize
from gpytorch.models import SingleTaskGP
from gpytorch.mlls import ExactMarginalLogLikelihood

# Create a sample dataset
train_X = torch.rand(10, 2, dtype=torch.double) * 2
# `train_Y` has a shape of torch.Size([10, 1]) due to keepdim=True
train_Y = 1 - (train_X - 0.5).norm(dim=-1, keepdim=True)
train_Y += 0.1 * torch.rand_like(train_Y)

# Initialize the GP model
gp = SingleTaskGP(
    train_X=train_X,
    train_Y=train_Y,
    input_transform=Normalize(d=2),
)

# Initialize the MLL
mll = ExactMarginalLogLikelihood(gp.likelihood, gp)

# Get the model output and compute the loss
output = gp(train_X)
loss = -mll(output, train_Y)

# My print statement shows 10 loss values
print(loss)
# Expected output: A single scalar value
# Actual output: A tensor with 10 values, e.g.,
# tensor([0.6178, 0.5820, 0.5738, 0.6130, 0.6213, 0.7199, 0.5811, 0.5917, 0.5738, 0.5961], dtype=torch.float64, grad_fn=&lt;NegBackward0&gt;)


</code></pre>
<p>I understand that the marginal log-likelihood should be a single value representing the overall model fit. Why is <code>gpytorch</code> producing a separate loss for each data point?</p>
<p>Thank you for your help.</p>
How Tensor.view works?

username: EricHermosis
cooked: <p>I’m looking into a more deep explanations about how Tensor.view works, looking into old forums it doesn’t allow non contiguous tensors, but now acording to documentation and what I’ve been playing it seems it does with some restrictions, however is not very clear to me how.</p>
<p>If I understood right there are two cases, view is a subspan of the original dimensions, that is (source.rank() &gt; view.rank()) or it expands the rank of the source (opossite case) if the view have more dimensions than the source then the following condition should be meet:</p>
<p>stride[i] == stride[i+1] * shape[i+1] for dimension &lt; rank to dimension = source.rank</p>
<p>However is not very clear to me how final strides are calculated for the rest of the view, since docs stops there and it confuzes me a bit.</p>

username: Arunprakash-A
cooked: <p>Hi <a class="mention" href="/u/erichermosis">@EricHermosis</a>, do check out this <a href="https://blog.ezyang.com/2019/05/pytorch-internals/" rel="noopener nofollow ugc">blog</a> to understand PyTorch’s internals, including the concepts of strides and views</p>
Torch.compile error on backward

username: richardzzj
cooked: <p>The compiling of forward worked just fine, but failed on backward:</p>
<pre data-code-wrap="bash"><code class="lang-bash">torch/_inductor/ir.py, line 2455, in _normalize_size:   
  assert len(new_size) == len(old_size)
torch._inductor.exc.LoweringException: AssertionError:
  target: aten.index_put.default
  args[0]: TensorBox(StorageBox(
    Pointwise(
      ‘cuda’,
      torch.float16,
      def innner_fn(index):
        _, i1, i2, i3, i4, i5 = index
        tmp0 = ops.constant(0, torch.float16)
        return tmp0
      ,
      ranges=[1, 239, 522, 8, 64, 2],
      origin_node=full_default,
      origins=OrderedSet([full_defult])
    )
  ))
  args[1]: [None, None, None, None, None, TensorBox(StorageBox(
    Pointwise(
      ‘cuda’,
      torch.int64,
      def inner_fn(index):
        tmp0 = ops.constant(1, torch.int64)
        return tmp0
      ,
      ranges=[],
      origin_node=lift_fresh_copy_12,
      origins=OrderedSet({lift_fresh_copy_12])
    )
  ))]
  args[2]: TensorBox(StorageBox(
    Pointwise(
      ‘cuda’,
      torch.float16,
      def inner_fn(index):
        _, i1, i2, i3, i4, _ = index
        tmp0 = ops.load(tangents_36, i4 + 64 * i3 + 512 * i2 + 267264 * i1)
        tmp1 = ops.load(tangents_29, i4 + 64 * i3 + 512 * i2 + 267264 * i1)
        tmp2 = ops.load(buf16, i4 + 64 * i3 + 512 * i2 + 267264 * i1)
        tmp3 = -tmp2
        tmp4 = tmp1 + tmp3
        tmp5 = ops.load(_tensor_constant14, i4 + 64 * i1)
        tmp6 = tmp4 * tmp5
        tmp7 = ops.to_dtype(tmp6, torch.float16, src_dtype=torch.float32)
        tmp8 = ops.load(tangents_39, i4 + 64 * i3 + 512 * i2 + 267264 * i1)
        tmp9 = ops.load(buf17, i4 + 64 * i3 + 512 * i2 + 267264 * i1)
        tmp10 = tmp8 + tmp9
        tmp11 = ops.load(_tensor_constant13, i4 + 64 * i1)
        tmp12 = tmp10 * tmp11
        tmp13 = ops.to_dtype(tmp12, torch.float16, src_dtype=torch.float32)
        tmp14 = tmp7 + tmp13
        tmp15 = tmp0 + tmp14
        return tmp15
      ,
      ranges=[1, 239, 522, 8, 64, 1],
      origin_node=add_28,
      origins=OrderedSet([add_28])
    )
  ))
  args[3]: True
</code></pre>
<p>It seems like the error is caused by torch.index_select:</p>
<pre data-code-wrap="python"><code class="lang-python">def forward(…):
  …
  x = x.reshape(1, 239, 522, 8, 64, 2)
  x = torch.index_select(x, dim=5, index=torch.tensor(0, dtype=torch.int64, device=‘cuda:0‘))
  x = x.squeeze(5)
  x = torch.mul(x, y)
  ...
</code></pre>
<p>I really not how to debug in backward while using torch.compile. Need some help to understand the reason of this error.<img src="https://discuss.pytorch.org/images/emoji/apple/face_with_medical_mask.png?v=14" title=":face_with_medical_mask:" class="emoji" alt=":face_with_medical_mask:" loading="lazy" width="20" height="20"></p>