akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Pytorch version

username: Yingshu-Li
cooked: <p>Hi</p>
<p>I’m currently working on projects on newer GPUs, such as the RTX 5090 that require CUDA 12.8. However, many of my dependencies and external packages are not compatible with the latest PyTorch releases (e.g., 2.7), and upgrading breaks a number of pipelines.</p>
<p>Unfortunately, earlier PyTorch versions like 2.3 do not provide wheels compatible with newer CUDA versions (e.g., CUDA 12.3 or 12.4). This causes difficulty for many users who:<br>
• Depend on stable older PyTorch versions for reproducibility or compatibility reasons;<br>
• Need to utilize newer GPUs (e.g., 5090) which require updated CUDA runtimes;</p>
<p>Are there any plans to provide updated builds of previous PyTorch versions (like 2.3 or 2.4) that support newer CUDA toolkits such as 12.8?<br>
In my experiments, the same project shows large discrepancies in training performance between PyTorch 2.3 and PyTorch 2.7 — in some cases, accuracy drops dramatically.<br>
Could you clarify what changes in the backend (e.g., kernels, precision behavior, compiler flags, etc.) might cause such differences?<br>
Is there a recommended way to debug these regressions?</p>

username: ptrblck
cooked: <p>No, already released binaries will not receive any updates and won’t be published again. I would recommend updating all dependencies allowing you to stick with the latest stable or nightly releases.</p>
Is it safe to do `torch.matmul(M, X, out=X)`?

username: Thomas_Ahle
cooked: <p>I would like to do <code>X = M @ X</code>, but without allocating an extra matrix on the RHS.<br>
So I tried <code>torch.matmul(M, X, out=X)</code> and it seems to work.</p>
<p>I can even do <code>torch.matmul(X, X, out=X)</code> and the results seem to come out right:</p>
<pre><code class="lang-auto">[ins] In [53]: x = torch.randn(4,4)

[ins] In [54]: torch.matmul(x, x)
Out[54]:
tensor([[ 7.7674,  2.4361,  1.5354, -1.0358],
        [ 4.4382, -1.2255, -0.2265,  0.7528],
        [-4.6821,  0.8569, -4.4141, -0.9833],
        [-1.8084,  6.9762,  2.8231,  1.0764]])

[ins] In [55]: torch.matmul(x, x, out=x)
Out[55]:
tensor([[ 7.7674,  2.4361,  1.5354, -1.0358],
        [ 4.4382, -1.2255, -0.2265,  0.7528],
        [-4.6821,  0.8569, -4.4141, -0.9833],
        [-1.8084,  6.9762,  2.8231,  1.0764]])
</code></pre>
<p>My question is whether it’s safe to rely on this behaviour? Or if I need to allocate a temporary tensor for the results, then copy them to X?</p>

username: eqy
cooked: <p>Generally this should be safe (if it wasn’t the op should complain about aliased pointers).</p>

username: Thomas_Ahle
cooked: <p>Well, it doesn’t work for <code>bmm</code>, and there are no complaints in that case:</p>
<pre><code class="lang-auto">[ins] In [57]: x = torch.randn(2,2,2)

[ins] In [58]: torch.bmm(x,x)
Out[58]:
tensor([[[ 0.2954,  0.6422],
         [-0.3119,  1.9941]],

        [[ 0.5595,  0.0932],
         [-0.4605,  1.6915]]])

[ins] In [59]: torch.bmm(x,x,out=x)
Out[59]:
tensor([[[-0.3908,  0.0000],
         [ 0.0000,  0.0000]],

        [[-0.1640,  0.0000],
         [ 0.0000,  0.0000]]])
</code></pre>
<p>Are you saying this should have raised an error?</p>

username: ngimel
cooked: <p>This should never be attempted as most likely it will produce wrong results. We are not checking that because that’s really a very niche case, and correctly writing a check that would be fast and accurate is hard, but it’s a very bad idea.</p>

username: Thomas_Ahle
cooked: <p>I’m not proposing using this to square matrices, I’m just trying to figure out if <code>out=</code> actually works at all, or if it just allocates an internal tensor and copies it to <code>out=</code> when it’s done.</p>
<p>Or are you suggesting to just never use the <code>out=</code> parameter?</p>
<p>Another weird thing: It seems that <code>out=temp</code> also works when <code>temp</code> has the wrong shape. This furthers my suspicion that maybe <code>out=</code> doesn’t actually work?</p>

username: shirazkn
cooked: <p>Does not seem to be safe, at least not anymore.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; I1 = torch.eye(3)
&gt;&gt;&gt; I2 = torch.eye(3) * 2
&gt;&gt;&gt; torch.matmul(I1, I2)
tensor([[2., 0., 0.],
        [0., 2., 0.],
        [0., 0., 2.]])
&gt;&gt;&gt; torch.matmul(I1, I2, out=I1)
tensor([[0., 0., 0.],
        [0., 0., 0.],
        [0., 0., 0.]])
</code></pre>
Symmetric parametrization

username: octave
cooked: <p>Hello,</p>
<p>From the PyTorch <a href="https://docs.pytorch.org/tutorials/intermediate/parametrizations.html#introduction-to-parametrizations" rel="noopener nofollow ugc">parametrization tutorial</a> the symmetric parametrization is implemented as follow:</p>
<pre><code class="lang-auto">class Symmetric(nn.Module):
    def forward(self, X):
        return X.triu() + X.triu(1).transpose(-1, -2)
</code></pre>
<p>Is there a reason (especially for gradient computation) why this would be preferred to:</p>
<pre><code class="lang-auto">class Symmetric(nn.Module):
    def forward(self, X):
        return X + X.transpose(-1, -2)
</code></pre>

username: KFrank
cooked: <p>Hi Octave!</p>
<aside class="quote no-group" data-username="octave" data-post="1" data-topic="220451" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/f05b48/48.png" class="avatar"> octave:</div>
<blockquote>
<p>Is there a reason (especially for gradient computation) why this would be preferred to:</p>
<pre><code class="lang-auto">class Symmetric(nn.Module):
    def forward(self, X):
        return X + X.transpose(-1, -2)
</code></pre>
</blockquote>
</aside>
<p>I can’t think of any compelling reason – both should work.</p>
<p>One thing to note is that the in the original version, the non-symmetric <code>X</code> matrix contains<br>
ignored elements – those in the lower triangle – while your version contains redundant<br>
elements (that aren’t ignored).  Arguably the former could be better.</p>
<p>In your case, you add two elements together – one from the upper triangle and one from<br>
the lower triangle.  So if you (or the optimization step) adds a large value to the upper<br>
element and subtracts the same value from the lower element, it would have no effect.<br>
This means that you have a directions (a “mode”) where those elements could drift off<br>
to <code>+-inf</code>, potentially causing problems.  (The ignored values in the original approach<br>
could also potentially drift off, but that’s less likely to happen and less likely to cause<br>
problems if they do.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: octave
cooked: <p>Hello K. Frank,</p>
<p>Thank you for all the details on how redundancy could cause problems!</p>
Re-initialise weights and biases to Pytorch default in k-fold cross-validation

username: NZ1
cooked: <p>I’ve defined a function that to optimise weights and biases so that i can run this every fold of a k-fold validation. However, i find that the default Pytorch weights and bias setting is better than my effort to improve on this. So how should i reset the Pytorch default weights and biases each time i pass through the next k fold, without resetting the whole model?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="NZ1" data-post="1" data-topic="220480">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/c57346/48.png" class="avatar"> NZ1:</div>
<blockquote>
<p>So how should i reset the Pytorch default weights and biases</p>
</blockquote>
</aside>
<p>You can call the <code>.reset_parameters()</code> method of the corresponding module to reset the trainable parameters using the default initialization.</p>
<aside class="quote no-group" data-username="NZ1" data-post="1" data-topic="220480">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/c57346/48.png" class="avatar"> NZ1:</div>
<blockquote>
<p>without resetting the whole model?</p>
</blockquote>
</aside>
<p>I don’t understand this part of the question as I would assume you want to (and should) reset the model in each new run.</p>

username: NZ1
cooked: <p>Thank you for responding. The weights and biases should be reset for each k-fold, otherwise they will be trained on the previous k-fold, meaning there will be data leakage, ie training on the test data. So i wrote a module to reset only the weights and biases, rather than reseting the whole model which might take longer. In resetting the weight and biases i just used general advice i found on the net, which was appropriate for the model. It’s a couple CNN layers followed by a couple of fully filled layers. However, just from trying to evaluate how good my weight and bias settings were i became aware that the default weights and bias setting by Pytorch was better than the efforts to do this. So what i was wanting to do was to somehow access the default Pytorch weights and bias settings for every k-fold loop, without resetting the model for each k-fold which would take longer. I’m still a bit of a novice in this area, but i hope this makes some sense. Many thanks,</p>
Profile a model layer by layer

username: Johannes_Vogt
cooked: <p>Hey, I find it difficult to understand wether it is possible to profile a pytorch layerwise.</p>
<p>So far, I wrap the layers in my model (nn.Module) using</p>
<pre><code class="lang-auto">with profiler.record_function("layer_name_shown_in_summary"):
     self.layer= SomeLayer(...)
</code></pre>
<p>I wrap the optimization loop in this expression</p>
<pre><code class="lang-auto">with torch.profiler.profile(
                activities=[
                    torch.profiler.ProfilerActivity.CPU,
                    torch.profiler.ProfilerActivity.CUDA,  # Optional: if using GPU
                ],
                record_shapes=True,
                with_stack=True,
                with_modules=True
            ) as prof:
       do the training here
</code></pre>
<p>and I print out the statistics using this comand here</p>
<pre><code class="lang-auto">print(prof.key_averages(group_by_input_shape=True).table(
        sort_by   = "self_cuda_time_total",
        row_limit = 20
        )
    )
</code></pre>
<p>The result looks like this</p>
<pre><code class="lang-auto">Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg     Self CUDA   Self CUDA %    CUDA total  CUDA time avg    # of Calls    Input Shapes
...
aten::mul         0.01%     245.862us         0.04%     899.349us      49.964us      57.740ms         3.98%      57.740ms       3.208ms            18     [[SomeBatchDim, SomeVectorDim], [SomeVectorDim]]
</code></pre>
<p>My question is:</p>
<ul>
<li>Is the time recorded layerwise?</li>
<li>Can I group and summarize the single function for each layer?</li>
</ul>
<p>Any help would be much appreciated.</p>

username: ptrblck
cooked: <p>The output should contain the <code>record_function</code> tag as also seen in the <a href="https://docs.pytorch.org/tutorials/recipes/recipes/profiler_recipe.html">profiler tutorial</a>.</p>
/home/system/miniconda3/envs/main/lib/python3.8/site-packages/torch/cuda/__init__.py:230: UserWarning: NVIDIA GeForce RTX 5070 Ti with CUDA capability sm_120 is not compatible with the current PyTorch installation. The current PyTorch install supports CU

username: Mahesh_Reddy_Tapeti
cooked: <p>Hey, I have rtx 5070 ti with ubuntu 24 and cuda 12.8. I got this error tried to reinstall but did’t work?  Whats the fix?</p>
<p>Thanks in advance</p>

username: ptrblck
cooked: <aside class="quote" data-post="5" data-topic="220460">
  <div class="title">
    <div class="quote-controls"></div>
    <img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/rtx-5070-ti-blackwell-pytorch-nightly-triton-still-getting-sm-120-is-not-defined-for-option-gpu-name-error/220460/5">RTX 5070 Ti (Blackwell) + PyTorch Nightly + Triton: Still Getting "sm_120 is not defined for option 'gpu-name'" Error</a> <a class="badge-category__wrapper " href="/c/windows/26"><span data-category-id="26" style="--category-badge-color: #3AB54A; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="This category is focused on PyTorch on Windows related issues."><span class="badge-category__name">windows</span></span></a>
  </div>
  <blockquote>
    Blackwell is already supported in nightly and the latest stable 2.7.0 release. Select the PyTorch binary from our install matrix with CUDA 12.8 and it will work.
  </blockquote>
</aside>
MAC inter processors cannot install torch2.5.1

username: tmac-weylin
cooked: <p>When I run a program, the program reports an error:<br>
AttributeError: module ‘torch.compiler’ has no attribute ‘is_compiling’.<br>
So I want to upgrade torch2.2.2 to 2.5.1, but I can’t install 2.5.1 with the pip command, error information:<br>
ERROR: Could not find a version that satisfies the requirement torch==2.5.1 (from versions: 2.0.0 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2)<br>
ERROR: No matching distribution found for torch==2.5.1</p>

username: ptrblck
cooked: <p>Which Python version are you using?</p>

username: kannangce
cooked: <p>I’m facing the same issue. I’m having <code>python 3.9.6</code>.</p>
<p>Moreover, I was trying to do the installation via <code>hatch</code> rather than the pip directly. But this works in ubuntu, the problem is just with the mac.</p>

username: kannangce
cooked: <p>Turned out that my <code>hatch</code> was using python 13. You can see that when you run<br>
<code>hatch  --verbose run app:main</code>.</p>
<ol>
<li>I had to delete the virtual environment used by hatch (shown in the response of command with <code>--verbose</code> option).</li>
<li>Restart things by using <code>python3.12</code>.</li>
</ol>

username: comsur
cooked: <p>I have the same problem. I have Python 3.12.2, MacOS.</p>

username: Naeem_Khoshnevis
cooked: <p>I encountered the same issue. I uninstalled Anaconda, deleted the cached Conda environment files, reinstalled Anaconda, and was then able to successfully install <code>torch 2.6.0</code>.</p>

username: Marcos_Pacheco_Jr
cooked: <p>I have the same issues trying to install torch2.6.0 on a Mac with Python 3.11.9</p>

username: ptrblck
cooked: <p>I don’t know if you are using an x86 system, but if so note that builds are not maintained anymore for these as described <a href="https://discuss.pytorch.org/t/why-no-macosx-x86-64-build-after-torch-2-2-2-cp39-none-macosx-10-9-x86-64-whl/204546/2">here</a>.</p>

username: Marcos_Pacheco_Jr
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, this seems to be the issue for me.</p>

username: Jing-Jing_Li
cooked: <p>This worked for me too! Apple silicon. Successfully upgraded to 2.7</p>
Why does this issue only occur on the 5090?

username: Yoonwoo_Ha
cooked: <p>I’m using the same PyTorch version 12.7.<br>
I’m training the same model on both the 3090 and the 5090 — there are no problems on the 3090, but on the 5090, the performance metrics suddenly drop significantly.<br>
I suspect the cause is the combination of a small value range and AMP.<br>
When I use a large value range (around 200 to 3000), AMP doesn’t cause any issues. However, when I normalize the values to a range of -1 to 1 or 0 to 1, the <code>train_loss</code> remains stable during training, but the <code>test_loss</code> suddenly spikes.<br>
If this also happened on the 3090, I’d assume it was a problem in my code, but since it only happens on the 5090 — and disappears when using a large value range — I think that’s the cause.<br>
I haven’t tested training without AMP yet because I can’t fit the model into VRAM without it.</p>
<p>How can I resolve this issue?</p>
visualize model performance

username: Kenza_Benjelloun
cooked: <p>Hello pytorch community. I am a beginner, and I have used nn.Sequential and nn.RNN to fit timeseries data I generated in a special form and I obtain different performance (both for mean square error and for mean absolute relative error). I would like to gather a maximum amount of plot ideas to realize either using pytorch packages (any recommendations?) or matplotlib / plotly to show the details of the performance of each model (I might add more in the future) in order to not only get a good idea of how well it performs but also the comparison in performance. Any other metric ideas would be greatly appreciated as well. Thank you.</p>

username: dbrenes
cooked: <p>Hello <a class="mention" href="/u/kenza_benjelloun">@Kenza_Benjelloun</a></p>
<p>When it comes to generating plots to compare metrics of different models, I would recommend the Weights&amp;Biases package. You can follow <a href="https://docs.wandb.ai/quickstart/" rel="noopener nofollow ugc">this guide</a> to get started with it.</p>
<p>Essentially, you’re can initialize it in your Python script like this:</p>
<pre><code class="lang-auto">run = wandb.init(
    project="timeseries",
    config={ 
        "model": "sequential"
    },
)
</code></pre>
<p>And then you can log data to <code>wandb</code>, which can be your metrics:</p>
<pre><code class="lang-auto">wandb.log({"mse": mse, "mae": mae})
</code></pre>
<p>This will automatically generate plots that are viewable in your <code>wandb</code> projects online, allowing you to compare results across different models and experiments, combine plots, and watch data being generated in real time.</p>
<hr>
<p><em>Machine Learning Engineer at <a href="https://www.ridgerun.ai/" rel="noopener nofollow ugc">RidgeRun.ai</a></em><br>
<em>Contact us: <a href="mailto:support@ridgerun.ai">support@ridgerun.ai</a></em></p>
FSDP hybrid sharding on multiple nodes

username: HariSeldon11988
cooked: <p>Hello together,</p>
<p>does it work in principle to run an FSDP training with hybrid_shard policy on a multinode setup, where each node has a different number of gpus?</p>
<p>So e.g. I have two nodes, first node has 8 GPUs and second node has 4 GPUs.</p>
<p>Thanks for any kind of help!</p>

username: H-Huang
cooked: <p>How would the sharding work in this case?</p>
<p>Node 1 (8 ranks) and node 2 (4 ranks) are replicated and they the model is sharded within each (imbalanced)? Or would you have 3 groups of 4 ranks (balanced)?</p>
<p>In theory, I think both would work but I don’t know if there is a way to express this in device_mesh (<a href="https://pytorch.org/tutorials/recipes/distributed_device_mesh.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with DeviceMesh — PyTorch Tutorials 2.6.0+cu124 documentation</a>). You may need to create the process groups yourself to express this.</p>
PyTorch faster than Libtorch for CNN inference

username: mamsds
cooked: <p>I narrow down the comparison to the below code snippets (<code>v16mms</code> is a List/Vector of models):</p>
<pre data-code-wrap="C++"><code class="lang-plaintext">spdlog::info("Running inference");
for (size_t i = 0; i &lt; v16mms.size(); ++i) {
  outputs[i] = v16mms[i].forward(input).toTensor();
  output += outputs[i];
}
spdlog::info("Done");
</code></pre>
<p>VS</p>
<pre data-code-wrap="Python"><code class="lang-plaintext">logging.info('Running inference')
for i in range(len(v16mms)):
    outputs.append(v16mms[i](images_tensor))
    output += outputs[i]
logging.info('Done')
</code></pre>
<p>Two snippets do the same thing, they run the same set of models and then add the inference results to <code>output</code>.</p>
<p>My tests show that PyTorch version takes ~230ms while the LibTorch version takes ~300ms. Any idea why LibTorch is even slower?<br>
(In case you want a minimally reproducible example, you can find the LibTorch file <a href="https://github.com/alex-lt-kong/vgg16-based-pipeline/blob/main/src/debug.cpp" rel="noopener nofollow ugc">here</a> and the PyTorch file <a href="https://github.com/alex-lt-kong/vgg16-based-pipeline/blob/main/src/debug.py" rel="noopener nofollow ugc">here</a>)</p>

username: gengxingze
cooked: <p>Firstly, if you are exporting the model via jit and then using it in c++ you should cycle the model derivation time a few more times before it stabilises, the initial few times are time consuming due to the need to build a graph<br>
example:<br>
Load model successful !<br>
t 277.576ms<br>
t 213.905ms<br>
t 187.971ms<br>
t 86.0545ms<br>
t 47.8444ms<br>
t 47.9361ms<br>
t 50.0757ms<br>
t 60.5665ms<br>
t 46.4637ms<br>
t 46.4323ms<br>
t 46.995ms<br>
t 47.0363ms<br>
Secondly, assignment between tensors in c++ is slow, if you want to speed up in c++ try using pointers.</p>

username: Thanh_Bao_Nguyen
cooked: <p>Hi!! Im interested in the Tensor assignment vs pointer. Could you give an example to show how passing pointers look like and how it is faster?</p>

username: gengxingze
cooked: <p>I’m sorry to be so late, but you can check out the content of this site where he goes into more detail about this. <a href="https://blog.csdn.net/weixin_41496173/article/details/117366210" rel="noopener nofollow ugc">Libtorch中tensor读写方法对比_libtorch tensor 逐个赋值-CSDN博客</a></p>
PyTorch Multiprocessing: Train only some parameters at some epochs

username: ameliatqy
cooked: <p>Hello there,</p>
<p>I have question about PyTorch multiprocessing - I noticed that it must update all model parameters for all epochs. I noticed that when I allow my process to update all parameters for all epochs, things work fine. But I have to selectively update only parts of my model for two reasons:</p>
<ol>
<li>I am training a RL solution and I need to update my value model every epoch while only updating my policy model every 3 epochs or so. So I can update my value model every epoch like normal but have to selectively update my policy model only after every 3 epochs or so.</li>
<li>I have millions of user embeddings to train so updating all parameters for every epoch is very time consuming - I only want to update the embeddings I am training on.</li>
</ol>
<p>When I only selectively update certain tensors, I get the following error:<br>
terminate called after throwing an instance of ‘gloo::EnforceNotMet’<br>
what():  [enforce fail at /pytorch/third_party/gloo/gloo/transport/tcp/pair.cc:446] op.preamble.length &lt;= op.nbytes. 28690396 vs 10575872</p>
<p>Is there a way to selectively update certain tensors at certain epochs? Is this even possible with torch multiprocessing or should I look at other options? Thank you for any help in advance!</p>

username: H-Huang
cooked: <p>Multiprocessing does not affect your training, its just responsible for starting up multiple processes. In your error message you are using Gloo, so you are using some distributed API, is it just send/recv?</p>
<p>It seems doable to selectively update. To freeze parameter weights you just have to set <code>param.requires_grad=False</code>. Accumulating gradients across epochs would also work calling <code>torch.autograd.backward(..., retain_graph=True)</code></p>
<p>Do you have an example script to repro? I’m not sure what distributed APIs you are using or what your model looks like.</p>

username: ameliatqy
cooked: <p>Yeah I am using PyTorch Distirbuted and used Gloo for a start (will need to use GPUs and was going to transfer to NCCL after I get this code working). I can’t paste my whole code but I share the problematic snippets. I am training both a value and policy model on multiple GPUs - I update the value model every iteration but the policy model every 3 iterations. It hangs on the first iteration because, I am not sure, but it looks like it was waiting for the policy gradients to be sent:</p>
<pre><code class="lang-auto">for epoch in num_epoch:
  ...
  value_model.backwards()
  if epoch &gt; 0 and epoch % 3 == 0:
    policy_model.backwards()

</code></pre>
<p>The second issue is that I have 65M user embeddings to train and it does not fit in the GPU. As such, I only shift the embeddings I am training onto the GPU for that batch. This doesn’t work as well as the model waits for the gradients of all the other embeddings to be sent to the main GPU.</p>
Dataset of uneven lengths

username: LionFrangoulis
cooked: <p>I am trying to create a model for some chemistry application, however the number of atoms in each sample will eventually be different.<br>
Assuming N datapoints, with A atoms each and a feature number per atom of F, so far my input is an (N,A) tensor of atom species, that is supposed to be encoded into a randomly initialised encoding that is later trained, resulting in a tensor (N,A,F). This I believe I can perform for various lengths by combining the first two dimensions, creating a list of trainable tensors (one for each element expected) and the results simply being a tensor containing the trainable tensors in the order according to the atoms. However, this essentially mixes the dimension that seperates samples and the dimension within one sample.</p>
<p>Is there a better way of performing these, without creating a bunch of zero tensors to unify the samples in size? (as proposed in <a href="https://discuss.pytorch.org/t/how-to-create-a-dataloader-with-variable-size-input/8278/2" class="inline-onebox">How to create a dataloader with variable-size input - #2 by smth</a> and many other posts)</p>

username: vdw
cooked: <p>At the end of the day, your tensors must be “full”, i.e., they cannot contain arrays of different lengths. Even if you create your own <code>Dataset</code> class that will returns a list of tensors like that, you would still need to convert them into a single full tensor before giving it to a network.</p>
<p>Padding with zeros is a common best practice and you can use <a href="https://docs.pytorch.org/docs/stable/generated/torch.nn.utils.rnn.PackedSequence.html" rel="noopener nofollow ugc"><code>PackSequence</code></a> to make the network ignore the padding. Alternatively, you can write your own <code>Sampler</code> to organize your dataset such data each batch only contains sequences of the same length; see <a href="https://discuss.pytorch.org/t/tensorflow-esque-bucket-by-sequence-length/41284/13">here</a>. I also have a more elaborate Jupyter <a href="https://github.com/chrisvdweth/selene/blob/master/notebooks/batch_processing_sequence_tasks.ipynb" rel="noopener nofollow ugc">notebook</a> that goes through this. Maybe useful.</p>

username: LionFrangoulis
cooked: <p>Padding with zeros is certainly the easiest approach, however due to non-lineraities and the bias added in each linear layer, they have non zero contributions to the “real” data. I guess one could create custom masks that make sure these terms stay zero?</p>

username: vdw
cooked: <p>Sure, padding should usually always come with loss masking; I actually just created a <a href="https://github.com/chrisvdweth/selene/blob/master/notebooks/masking_sequence_models.ipynb" rel="noopener nofollow ugc">notebook</a> for that.</p>
Vmap runtime error

username: zengkaipeng
cooked: <p>I tried to optimize the process of derivative calculation in PyTorch using vmap. Here’s my original code:</p>
<pre><code class="lang-auto">dx_over_dt = torch.zeros_like(new_confs)
d2x_over_dt2 = torch.zeros_like(new_confs)
grad_mask = torch.ones(ts.shape[0]).to(device)
for i in range(num_atoms):
    for j in range(3):
        dx_over_dt[:, i, j] = torch.autograd.grad(
            new_confs[:, i, j], ts, create_graph=True,
            retain_graph=True, grad_outputs=grad_mask
        )[0]
        d2x_over_dt2[:, i, j] = torch.autograd.grad(
            dx_over_dt[:, i, j], ts, create_graph=True,
            grad_outputs=grad_mask
        )[0]
</code></pre>
<p><code>new_confs</code> is the output of my model, and <code>ts</code> is one of the inputs to my model with <code>ts.requires_grad = True</code>. Then I modified the code to:</p>
<pre><code class="lang-auto">batch_size = ts.shape[0]
new_confs = new_confs.reshape(batch_size, -1).contiguous()
grad_mask = torch.eye(new_confs.shape[-1]).to(new_confs)
grad_mask = grad_mask.repeat(batch_size, 1, 1)

dx_over_dt = vmap(lambda v: torch.autograd.grad(
    new_confs, ts, create_graph=True,
    retain_graph=True, grad_outputs=v
)[0], in_dims=1, out_dims=1)(grad_mask)

d2x_over_dt2 = vmap(lambda v: torch.autograd.grad(
    dx_over_dt, ts, create_graph=True, grad_outputs=v
)[0], in_dims=1, out_dims=1)(grad_mask)

dx_over_dt = dx_over_dt.reshape(batch_size, -1, 3)
d2x_over_dt2 = d2x_over_dt2.reshape(batch_size, -1, 3)
new_confs = new_confs.reshape(batch_size, -1, 3)
</code></pre>
<p>However, no matter how I adjust the model structure and format, this code will definitely throw an error on the third call (the code successfully runs for two batches) with the following error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/ee490d8b3a8459393f3bf2bfd0c48846e7807516.png" data-download-href="https://discuss.pytorch.org/uploads/default/ee490d8b3a8459393f3bf2bfd0c48846e7807516" title="1748531453676"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee490d8b3a8459393f3bf2bfd0c48846e7807516_2_690x197.png" alt="1748531453676" data-base62-sha1="xZY9uuWNPbJHAEgFexueY2NckXc" width="690" height="197" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee490d8b3a8459393f3bf2bfd0c48846e7807516_2_690x197.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee490d8b3a8459393f3bf2bfd0c48846e7807516_2_1035x295.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/ee490d8b3a8459393f3bf2bfd0c48846e7807516_2_1380x394.png 2x" data-dominant-color="216267"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1748531453676</span><span class="informations">1769×507 269 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I would like to know why this happens. Is this a bug in vmap, or is there an alternative way to write this code?</p>
Pytorch doesnt support sm 120 5070 ti

username: Muhammad_Umar_Shahza
cooked: <p>Hey i really need help i got the top tier pc made it has 5070 ti when i try to run some tts as they work on pytorch it doesnty suport the gpu and error occur like it doesnt support yet is there any unofficial version that supporting 5070 ti? so i can run those tts ???</p>

username: ptrblck
cooked: <p>PyTorch binaries support Blackwell GPUs in the nightly builds as well as in the latest stable release. You need to install the binary with CUDA 12.8 runtime dependencies.</p>
_scaled_dot_product_efficient_attention implementation

username: arunava5764
cooked: <p>I was profiling a ViT model using pytorch profiler. It shows “_scaled_dot_product_efficient_attention” function called 12 times. I want to see the implementation of this function in pytorch github repo. The function name is mentioned in line no 766 of the following link:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/transformers/attention.cpp">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/transformers/attention.cpp" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/transformers/attention.cpp" target="_blank" rel="noopener nofollow ugc">aten/src/ATen/native/transformers/attention.cpp</a></h4>

<div class="git-blob-info">
  <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/transformers/attention.cpp" rel="noopener nofollow ugc"><code>main</code></a>
</div>


      <pre><code class="lang-cpp">#include &lt;ATen/core/TensorBody.h&gt;
#define TORCH_ASSERT_ONLY_METHOD_OPERATORS
#include &lt;ATen/core/Tensor.h&gt;
#include &lt;ATen/TensorOperators.h&gt;
#include &lt;ATen/AccumulateType.h&gt;
#include &lt;ATen/Dispatch.h&gt;
#include &lt;ATen/OpMathType.h&gt;
#include &lt;ATen/mps/MPSDevice.h&gt;
#include &lt;ATen/native/DispatchStub.h&gt;
#include &lt;ATen/NestedTensorImpl.h&gt;
#include &lt;ATen/TensorIndexing.h&gt;
#include &lt;ATen/TensorSubclassLikeUtils.h&gt;
#include &lt;ATen/native/transformers/attention.h&gt;
#include &lt;ATen/native/transformers/sdp_utils_cpp.h&gt;
#include &lt;c10/util/typeid.h&gt;
#include &lt;c10/core/DeviceType.h&gt;
#include &lt;c10/core/SymInt.h&gt;
#include &lt;c10/core/SymIntArrayRef.h&gt;
#include &lt;c10/util/Logging.h&gt;
#include &lt;c10/core/DispatchKey.h&gt;
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/main/aten/src/ATen/native/transformers/attention.cpp" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Also, in the same file the header file associated with this function is included in line no 40. But I cannot find any file where the function is implemented. Same goes for “_efficient_attention_forward” function showing in the profiler output. Is there any way to see those implementation in pytorch github?</p>
Torch.compile usage for generative models

username: ddilbazTT
cooked: <p>What is the best way to use <code>torch.compile</code> for generative models (i.e. invoking model.generate)?</p>
<p>Assuming the following code:</p>
<pre><code class="lang-auto">import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
AutoTokenizer.from_pretrained(
    "microsoft/phi-2", torch_dtype=torch.bfloat16
)
AutoModelForCausalLM.from_pretrained("microsoft/phi-2")
input_str = '''def print_prime(n):
            """
            Print all primes between 1 and n
            """'''
inputs = tokenizer(input_str, return_tensors="pt", return_attention_mask=False, max_length=128)
results = model.generate(inputs)
decoded_output = tester.tokenizer.decode(results[0])
</code></pre>
<p>When does one use <code>torch.compile</code>? I am using my own backend, so I noticed the following:</p>
<pre><code class="lang-auto">model = torch.compile(model, backend=my_backend...)
outputs = model.generate(inputs)
</code></pre>
<p>doesn’t actually run on my backend.<br>
I could do either of the following though:</p>
<pre><code class="lang-auto">model.forward = torch.compile(model.forward, backend=my_backend...)
outputs = model.generate(inputs)
</code></pre>
<p>or</p>
<pre><code class="lang-auto">fnc = torch.compile(model.generate, backend=my_backend...)
outputs = fnc(inputs)
</code></pre>
<p>My question is - what’s the best practice for using torch.compile for generative models? What’s the tradeoff between each ways to use <code>torch.compile</code>?</p>

username: ptrblck
cooked: <p>You could add the decorator to the method you would like compile via:</p>
<pre data-code-wrap="python"><code class="lang-python">@torch.compile
def generate(...):
   ...
</code></pre>

username: ddilbazTT
cooked: <p>Thanks for the reply!<br>
I noticed that my model gets killed due to out-of-memory when running with</p>
<pre><code class="lang-auto">model.forward = torch.compile(model.forward, backend=my_backend...)
outputs = model.generate(inputs)
</code></pre>
<p>But it can run when I do</p>
<pre><code class="lang-auto">fnc = torch.compile(model.generate, backend=my_backend...)
outputs = fnc(inputs)
</code></pre>
<p>Is this expected?</p>

username: ptrblck
cooked: <p>I assume your process gets killed from the OS as you are running out of memory on the host?</p>

username: ddilbazTT
cooked: <p>Yes exactly, so I was wondering if there is a recommended way of compiling generative models, or common effects to be aware of.</p>

username: ptrblck
cooked: <p>You could check if reducing the number of threads via the env variable <code>TORCHINDUCTOR_COMPILE_THREADS</code> could help avoiding the OOM.</p>
Torch Profiler for Multi-process Profiling?

username: Outage4778
cooked: <p>Hi, I am wondering if it is possible for the torch.profiler to profile multiple processes at once on a single machine. I would like to produce a chrome trace where there are different rows for different processes that are executing. I have seen the <a href="https://pytorch.org/tutorials/recipes/distributed_rpc_profiling.html" rel="noopener nofollow ugc">profiler RPC tutorial</a>, but this does not meet my needs as I do not use RPC since I am only using a single machine. Is it possible to produce traces like shown in the tutorial but without using RPC?</p>

username: longtran1904
cooked: <p><a class="mention" href="/u/outage4778">@Outage4778</a> It seems late but I’m also looking for the this method.<br>
I’m studying how much memory this Multi-processing approach costs for one machine.</p>
Help with pytorch running on Pop_OS!

username: Alikebrahim
cooked: <p>I have been trying to get pytorch to recognize my gpu on Pop_OS! for over a month now with no success. I’m trying to get started learning ML/AI as a beginner but have been facing this hurdle for a while now.<br>
I’ve looked into many threads that seemed relevant on this forum, and many other places. I’ve tried debugging the issue with the help of AI, too, to no avail. I’m having this issue both on my laptop (Nvidia 4070) and desktop (Nvidia 3070), both running Pop_OS and the latest nvidia driver provided by system76 (570.133.07).<br>
When trying to check if torch recognizes the gpu I get:</p>
<blockquote>
<p>Blockquote</p>
<blockquote>
<blockquote>
<p>print(torch.cuda.is_available())<br>
/home/alikebrahim/ComfyUI/.venv/lib/python3.12/site-packages/torch/cuda/<strong>init</strong>.py:174: UserWarning: CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu. (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:109.)<br>
return torch._C._cuda_getDeviceCount() &gt; 0<br>
False<br>
Can someone help guide me how to debug this and sort it out, please. I’ve tried with different versions of python, torch and even used different venv initializers (uv, virtualenv and conda)!</p>
</blockquote>
</blockquote>
</blockquote>

username: ptrblck
cooked: <p>The error message points to an error in the driver initialization:</p>
<pre data-code-wrap="python"><code class="lang-python">CUDA initialization: CUDA driver initialization failed, you might not have a CUDA gpu.
</code></pre>
<p>which is usually caused by a broken NVIDIA driver installation and unrelated to PyTorch.<br>
You might want to reinstall the driver and make sure any simple CUDA sample can be executed.</p>

username: Alikebrahim
cooked: <p>Thanks for your reply. But re-installing the driver doesn’t work. I’ve tried it and I get the same issue. Not sure what’s the problem honestly. By a stroke of luck, I worked on my desktop, but I still am facing the same issue on my laptop.</p>
Understanding torch.vmap with torch.channels_last_3d

username: michailb
cooked: <p>Hello everyone,</p>
<p>I stumbled across the following when playing around with <code>torch.vmap</code> in torch version <code>2.4.0</code>. Consider the following complete code snippet:</p>
<pre><code class="lang-auto">import torch

func = torch.vmap(lambda x, kernel: torch.nn.functional.conv3d(x, kernel, padding="same"), in_dims=(0, None))

data = torch.rand(16, 16, 16, 32, 32, 32) # [Group, Batch, Feature, X, Y, Z]
kernel = torch.rand(1, 16, 3, 3, 3) # [OutFeature, InFeature, X, Y, Z]
res = func(data, kernel)

print(res.shape) # torch.Size([Group, Batch, OutFeature, X, Y, Z])
</code></pre>
<p>This snippet works as expected, and the yielded shape is <code>[16, 16, 1, 32, 32, 32]</code>. Lets say now that we want to try doing this convolution using channels last to improve memory access patterns.<br>
We change the lambda within the <code>vmap</code> as follows:</p>
<pre><code class="lang-auto">func = torch.vmap(lambda x, kernel: torch.nn.functional.conv3d(x.to(memory_format=torch.channels_last_3d), kernel, padding="same"), in_dims=(0, None))
</code></pre>
<p>I thought that this would work, but it actually causes <code>vmap</code> to throw the following:</p>
<pre><code class="lang-auto">RuntimeError: required rank 5 tensor to use channels_last_3d format
</code></pre>
<p>This leads me to believe that I do not understand either how <code>vmap</code> or how <code>channels_last_3d</code> works. In my mind the tensor should be 5D since this operation runs for every <code>Group</code>. In fact, printing the tensor’s shape within the function gives a 5D shape. Furthermore, I see no reason for <code>channels_last_3d</code> not to work, as the memory is contiguous and (as I understand it) the memory format casting doesn’t change anything, it just passes information down to select which convolution algorithm will be used.</p>
<p>The obvious workaround in my case is to flatten the two batch dimensions, which I can work with. But it would be interesting to know what is going on regardless.</p>
<p>Thank you very much for your time,<br>
Mike</p>

username: ptrblck
cooked: <p>I can reproduce this using the latest nightly binary. CC <a class="mention" href="/u/richard">@richard</a> would you know why the rank 5 tensor is not detected?</p>
Why do we implement models like Llama again in examples/models/ and how is it different from the HF implementation for the same?

username: anzr299
cooked: <p>Hi everyone,</p>
<p>I noticed that in executorch, the models are also implemented in <code>examples/model/</code> which are used for exporting the model to ExportedProgram. But my question is how is this model implementation different from HF implementation and also the optimum-executorch flow for exporting the models.</p>

username: anzr299
cooked: <p>This question is answered here: <a href="https://github.com/pytorch/executorch/issues/11130" class="inline-onebox" rel="noopener nofollow ugc">Why are models like Llama implemented again in examples/models/ ? · Issue #11130 · pytorch/executorch · GitHub</a></p>
torch::nn::DistributedDataParallel C++ class?

username: mEm
cooked: <p>Hi everyone, I was wondering: why isn’t the <code>DistributedDataParallel</code> a C++ core class, while it seems to be a Python only feature?</p>

username: yf225
cooked: <p>I believe <code>DistributedDataParallel</code> theoretically can be exposed as part of C++ frontend. We are open to contribution PR if you are interested in contributing.</p>

username: mEm
cooked: <p>I understand. Can you just confirm that as per the current status, DDP is not available from the C++ frontend?<br>
I can definitely consider contributing, though for the moment the simplest solution was to upstream my changes up to the Python frontend.</p>

username: H-Huang
cooked: <aside class="quote no-group" data-username="mEm" data-post="3" data-topic="220283">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/90ced4/48.png" class="avatar"> mEm:</div>
<blockquote>
<p>Can you just confirm that as per the current status, DDP is not available from the C++ frontend?</p>
</blockquote>
</aside>
<p>Correct, there is no C++ frontend for DDP. DDP mainly leverages reducer (<a href="https://github.com/pytorch/pytorch/blob/main/torch/csrc/distributed/c10d/reducer.cpp" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/csrc/distributed/c10d/reducer.cpp at main · pytorch/pytorch · GitHub</a>) which is written in C++, but we have no explicit API for DDP at the C++ level.</p>
What's the difference between running compiled_model directly and running the captured graph module?

username: Huiyao
cooked: <p>In our case, we need to return different graphs to run according to different application scenarios. In order to avoid the dynamo cache returning the same graph every time, we choose to let the custom compiler return an empty graph and save the differently optimized graphs, and then call the specific graphs outside compiler like this:</p>
<pre data-code-wrap="python"><code class="lang-python"># 1. In compiler:
def __call__(original_gm: torch.fx.GraphModule, example_inputs):
    ...
    compiler.fwd_gm = original_gm
    return get_none_module(original_gm, self.variable_pool, self.alias_analysis)

# 2. run real graph module
 output = compiler.run_gm(compiler.fwd_gm)
</code></pre>
<p>However, it just return some tuple-type values rather than the expected dict-type values. Do you know  how to make the return value as expected the same as running compiled model directly?</p>
CNN output explosion

username: Nyx
cooked: <p>Hi,<br>
I’m training a CNN model using a TripletMarginLoss with no FC layers, when I print the output of the flatten layer after a few epochs I find that the output values explodes (e.g. 1e+23) what could be the reason for that and how to fix it?</p>
<p>I’m using ReLU activation function and Adam optimizer</p>
<pre><code class="lang-auto">class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.layers = []
        self.layers.append(nn.LazyConv2d(out_channels=8, kernel_size=1, stride=1))
        for i in range(cfg.BLOCKS_NUMBER):
            if i == 0: 
                self.layers.append(nn.LazyConv2d(out_channels=16, kernel_size=5, padding=2, stride=1))
                self.layers.append(nn.ReLU())
                self.layers.append(nn.LazyConv2d(out_channels=16, kernel_size=5, padding=2, stride=1))
                self.layers.append(nn.ReLU())
                self.layers.append(nn.LazyConv2d(out_channels=16, kernel_size=5, padding=2, stride=1))
                self.layers.append(nn.ReLU())
            else: 
                self.layers.append(nn.LazyConv2d(out_channels=256, kernel_size=3, padding=1, stride=1))
                self.layers.append(nn.ReLU())
                self.layers.append(nn.LazyConv2d(out_channels=256, kernel_size=3, padding=1, stride=1))
                self.layers.append(nn.ReLU())
                self.layers.append(nn.LazyConv2d(out_channels=256, kernel_size=3, padding=1, stride=1))
                self.layers.append(nn.ReLU())
            
            self.layers.append(nn.MaxPool2d(kernel_size=2, stride=2, padding=1))
        
        self.layers.append(nn.Flatten())
        self.model = nn.Sequential(*self.layers)
        


    def forward(self, anchors, positives, negatives):
        
        a = self.model(anchors)
        p = self.model(positives)
        n = self.model(negatives)

        return a, p, n
</code></pre>
<p>I also have parameter initialization as follows:</p>
<pre><code class="lang-auto">model = Model()

model.to(cfg.DEVICE)

input = torch.randn((1, 1, 640, 560)).to(cfg.DEVICE)
output1, output2, output3 = model(input, input, input)
model.apply(init_weights)
</code></pre>

username: ajayrkumar
cooked: <p>Short answer is the absence of Normalization layers.</p>
<p>As <code>cfg.BLOCKS_NUMBER</code> increases, your model grows in size, with a lot of convolution layers stacked one above the other. This is the most common reason for gradient explosion in CNNs.<br>
Without normalization, the activations can grow rapidly as they pass through each layer, especially with ReLU, which is unbounded above.</p>
<p>The solution to the exploding gradient problem is to use <a href="https://docs.pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html" rel="noopener nofollow ugc"><code>BatchNorm2D</code> layers</a>.</p>
<p>Follow <code>Conv-BatchNorm-ReLU</code> in your implementations to avoid this issue.</p>