akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch.multiprocessing: pass metadata or class wrapper for shared memory CUDA tensor?

pierisk: <p>I’m using <a href="https://pytorch.org/docs/stable/multiprocessing.html#sharing-cuda-tensors" rel="noopener nofollow ugc">this documentation guide</a> to share a CUDA tensor between 2 python mp processes using <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.share_memory_.html#torch.Tensor.share_memory_" rel="noopener nofollow ugc">share_memory_()</a> and a torch.mp.Queue.<br>
The tensors in the source arrive from an external source in streams as CUDA buffers, so I don’t have full control apart from converting them to a Tensor allocated on CUDA device (if you want details I can provide them in a reply).</p>
<p>I wanted to pass some <strong>metadata</strong> along with the CUDA tensor e.g. a string name/key for each image in the batch tensor (i.e. the first dimension indices of the tensor).<br>
I was wondering if the same shared memory (zero copy) between processes can work when:</p>
<ol>
<li>Wrapping the CUDA tensor to be shared in a python class with these metadata as additional attributes.</li>
<li>Somehow attaching metadata within the tensor itself. I saw <a href="https://dev-discuss.pytorch.org/t/is-there-a-place-for-storing-custom-data-within-pytorch-tensor/1085/8" rel="noopener nofollow ugc">this topic discussion</a> but it seems non-standard and non-trivial at least.</li>
</ol>
<p>Is there a way to send metadata along with each tensor in any way?<br>
Thank you very much for all your help!</p>
Tensor subtraction

Dirk10001: <p>Hello all,</p>
<p>I want to subtract tensor1 [1,384] from tensor2 [ 1, 384, 56, 56]. The images of tensor2 should be subtracted by the scalar in tensor1.</p>
<pre><code class="lang-auto">result= tensor2 - tensor1
result= at::sub(tensor2-tensor1)
</code></pre>
<p>Both failed. I do not want to blow tensor1 to [1,384,56,56].<br>
Many thanks for your help.</p>
<h1><a name="p-464361-tensor1view-1-doesnt-work-1" class="anchor" href="#p-464361-tensor1view-1-doesnt-work-1"></a>tensor1.view(-1) doesn’t work</h1>

Dirk10001: <p>Tensor tensor1flat= tensor1.view(-1)<br>
Tensor tensor2flat.view{384,56,56})</p>
<p>Doesn’t work.</p>

J_Johnson: <p>Try this:</p>
<pre><code class="lang-auto">import torch

#starting conditions
a = torch.rand((1,384))
b = torch.rand((1, 384, 56, 56))

#modify
a = a.unsqueeze(-1).unsqueeze(-1)

#operation
c = b - a
</code></pre>

Dirk10001: <p>Hello J_Johnson,</p>
<p>thank you, works fine.</p>
<pre><code class="lang-auto">a= a.unsqueeze(-1).unsqueeze(-1)
</code></pre>
<p>changes (1,384) to (1,384,1,1)</p>
FlexAttention customizability

veritas: <p>Hello. I am trying to implement the following 2 features and I am curious if it is possible with FlexAttention.</p>
<ol>
<li>I would like to use a different value for softmax numerical stabilization. Currently, softmax uses exp(value - max_value) for numerical stability. However, this is inefficient because we have to do two memory accesses: once to find the max value and once to get the sum. If the max_value can be set to a constant, this would make the implementation simpler. Also, combined with soft clipping with tanh, there would also be no overflow.</li>
<li>Would it be possible to implement Softmax1 from “Attention is off by 1”? I am aware it does not have much effect in practice, but I dislike using an incorrect equation.</li>
</ol>
Setting max_split_size_mb

tpg4876: <p>Hi all,<br>
I have a question regarding the usage of max_split_size_mb. On a technical level, I don’t understand how setting this parameter reduces external memory fragmentation.<br>
My understanding is that Pytorch allocates memory blocks dynamically rather than allocating fixed-size blocks. In that sense, isn’t it better to have higher values for max_split_size_mb, as that reduces the risk of lots of small blocks being allocated, with unusable blocks in between. However, if there is no default maximum value for max_split_size_mb, then surely this variable is already optimised for the type of memory allocation used by Pytorch?<br>
Thanks</p>
Feed a model with cumulative sum of sampled classified sequences

Seam1: <p>Hi guys,</p>
<p>I would like to implement a model using as input features cumulative sums of classes resulting from sampled sequences classification.</p>
<p>Let’s imagine we want to build signals (hourly or daily) which would be the sums of classified sequences of text, into a fix number of classes. And we want to feed a larger model with this multivariate feature. So in forward method we should iterate over sequences to classify it and build resulting sums of classes for each time steps.</p>
<p>What would be the most efficient way to implement this ? Ideally, I would like the last linear layer from the seq2class model to be trainable.</p>
Matmul mixed dtypes

Evgeni_Burovski: <p>Currently, matrix multiplication seems to require matching dtypes of its inputs:</p>
<pre><code class="lang-auto">In [6]: torch.ones(3, dtype=torch.float32) @ torch.ones(3, dtype=torch.float64)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
Cell In[6], line 1
----&gt; 1 torch.ones(3, dtype=torch.float32) @ torch.ones(3, dtype=torch.float64)

RuntimeError: dot : expected both vectors to have same dtype, but found Float and Double
</code></pre>
<p>Is it a deliberate decision (to limit the number of kernels or some such), or is it an oversight which can be fixed in the future?</p>
<p>The context for the question is the array API compatibility work in SciPy, <a href="https://github.com/scipy/scipy/pull/22008#discussion_r1876759282" class="inline-onebox" rel="noopener nofollow ugc">ENH: signal.vectorstrength: add array API standard support by ev-br · Pull Request #22008 · scipy/scipy · GitHub</a>. A workaround is not particularly difficult though; whether a workaround needs to be permanent or temporary, would be great to have a canonic answer for, hence this post :-).</p>
<p>Cheers,</p>
<p>Evgeni</p>
How to load the best weights of yolov6 after training?

edgy_sharma: <p>import torch<br>
from ultralytics import YOLO<br>
from yolov6.models.yolo import build_model</p>
<p>device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)</p>
<p>model_path = ‘path_to_weights.pt’</p>
<p>model = build_model(cfg=‘/path/yolov6n.yaml’, num_classes=1, device=device, fuse_ab=False, distill_ns=False)</p>
<p>model.torch.load(model_path)</p>
<p>model.eval()<br>
.<br>
.<br>
.</p>
<p>I am performing object detection and I have trained my dataset on yolov6. Now, I want to use the best weights learned during training to perform another task but I am facing difficulty in loading these weights. Can you help me find the correct way to do this?</p>

ptrblck: <p>Could you describe what kind of errors you are seeing while trying to load the weights?<br>
A minimal example of saving and loading the parameters and buffers would be:</p>
<pre data-code-wrap="python"><code class="lang-python">sd = model.state_dict()
torch.save(sd, "checkpoint.pt")
...
sd = torch.load("checkpoint.pt")
model = MyModel()
model.load_state_dict(sd)
</code></pre>
Pytorch on 3.10 python version of jetson orin

Vaibhavi_Kavathekar: <p>I have two versions of python on my jetson orin. One is 3.8 and the other is 3.10 which is installed in a virtual enviornment. My cuda version is 11.4 and I was trying to install pytorch cpu version 2.5.1 but was unable to. I did try this - ```<br>
pip install torch==2.5.0 torchvision==0.20.0 torchaudio==2.5.0 --index-url <a href="https://download.pytorch.org/whl/cpu" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cpu</a></p>
<pre data-code-wrap="it"><code class="lang-it"></code></pre>

ptrblck: <p>You would need to install the binaries from e.g. <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048">here</a>.</p>

Vaibhavi_Kavathekar: <p>Okay. and according to the link I need to have jetpack 6.0 for python 3.10 right? Currently my jetpack version is 5.1</p>
Wrapping a DDP module inside a simple Module

mr-mapache: <p>Let’s say I want to create a model agnostic wrapper like this:</p>
<pre data-code-wrap="python"><code class="lang-python">from torch import Tensor
from torch.nn import Module

class Classifier(Module):
          def __init__(self, model: Module, criterion: Module, optimizer: Optimizer):
              super().__init__()
              self.epoch = 0
              self.model = model
              self.criterion = criterion
              self.optimizer = optimizer

          def forward(self, input: Tensor) -&gt; Tensor:
              return self.model(input)
          
          def loss(self, output: Tensor, target: Tensor) -&gt; Tensor:
              return self.criterion(output, target)

          def fit(self, input: Tensor, target: Tensor) -&gt; tuple[Tensor, Tensor]:
              self.optimizer.zero_grad()
              output = self(input)
              loss = self.loss(output, target)
              loss.backward()
              self.optimizer.step()
              return output, loss

          def evaluate(self, input: Tensor, target: Tensor) -&gt; tuple[Tensor, Tensor]: 
              output = self(input)
              loss = self.loss(output, target)
              return output, loss
</code></pre>
<p>However let’s say I need distributed training. How should I approach to this with distributed data parallel?  I found that data parallel don’t work well with custom methods in a module, however here I can pass a ddp module instance instead of a simple module for my classifier’s main nn, will this be ok? Or should i make my whole classifier a ddp? or should I remove the inheritance from Module in Classifier?</p>
ImportError while installing mamba-ssm

Omaralmaqtari: <p>I am trying to install mamba-ssm package through <code>pip install mamba-ssm</code> but I got the following error:</p>
<pre><code class="lang-auto">ImportError: /tmp/pip-build-env-wl_dvsa4/overlay/lib/python3.12/site-package/torch/lib/../../nvidia/cusparse/lib/libcusparse.so.12: symbol __nvJitLinkComplete_12_4 version libnvJitLink.so.12 not defined in file libnvJitLink.so.12 with link time reference
</code></pre>
<p>The last import before the error: <code>from torch._C import *</code>, even tho the same import works fine when I call it separately.<br>
Then I tried to install it through downloading the github package and running the setup.py file, but I got another error (older version of GCC, you need GCC 9 or newer).</p>
<p>my setup contains the following:<br>
x86 PC with nvidia titan GPU<br>
ubuntu 18.04<br>
cudatoolkit 12.0<br>
python 3.12<br>
pytorch 2.5.1<br>
pytorch-cuda 12.1</p>
<p>I am really lost now, Any Help Would be Appreciated.</p>

ptrblck: <aside class="quote no-group quote-modified" data-username="Omaralmaqtari" data-post="1" data-topic="215907">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/omaralmaqtari/48/72219_2.png" class="avatar"> Omaralmaqtari:</div>
<blockquote>
<p>cudatoolkit 12.0<br>
…<br>
pytorch-cuda 12.1<br>
…<br>
libcusparse.so.12: symbol __nvJitLinkComplete_12_4<br>
…</p>
</blockquote>
</aside>
<p>It seems you are trying to mix various CUDA libs from 12.0, 12.1, and 12.4 and I would recommend sticking to a single one.</p>

Omaralmaqtari: <p>Thank you so much for your help.</p>
<p>I did not intend to install different CUDA libs, I installed CUDA toolkit 12.0, but then there wasn’t a similar version for pytorch-cuda, so I installed pytorch-cuda 12.1 instead. For the libcusparse 12.4, it was automatically installed by the <code>pip install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia</code>.<br>
Anyway for future readers, the different CUDA libs was not the problem in my case, I solved the problem by updating the GCC and G++ libs in my ubuntu OS from 7.5. into 9.0 using the following commands:</p>
<pre><code class="lang-auto">sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt update
sudo apt install gcc-9
sudo apt install g++-9
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 60 --slave /usr/bin/g++ g++ /usr/bin/g++-9
</code></pre>
<p>then I installed the mamba-ssm package locally by downloading the GitHub repo and opening a terminal in the same folder and <code>python setup.py install</code> command.</p>
<p>I hope this post helps.</p>
InfiniBand Vs TCP

FGeraci: <p>Hello,</p>
<p>I am comparing the same workflow accross InfiniBand (IB) and TCP. the workflow is one of the first example from the torch tune repo.</p>
<p>IB compute time 37.8min <code>NCCL_IB_DISABLE=0</code><br>
TCP compute time 32min <code>NCCL_IB_DISABLE=1</code></p>
<p>Should IB be faster than TCP?</p>
<p>If needed I can add the yaml config</p>

FGeraci: <p>adding to my previous post.</p>
<p>I run several experiments comparing runtime, between jobs using IB and jobs not using IB. the tests are run on the same hardware, 2 node with 8 h100 each.</p>
<p>I noticed that GPUs start being utilized after 621sec for NO_IB and 1081sec for IB.</p>
<p>the test is a fine-tuning a 7B model using torchtune.</p>
Loss calculation error

BarMar: <p>Hi,<br>
I want to implement post-hoc auditing of LLMs (AutoModelForCasualLM) (specifically Pythia-1b), and I would need access to gradients at each step (actually gradient per sample, but other methods I found here causes problems, e.g. torch.all in the transformers library). However, while calculate a loss I got the following problem<br>
AttributeError: ‘Parameter’ object has no attribute ‘_forward_counter’.</p>
<pre><code class="lang-auto">input_ids = sample['input_ids'].unsqueeze(0).to(self.device)
attention_mask = sample['attention_mask'].unsqueeze(0).to(self.device)
loss = model(input_ids=input_ids, attention_mask=attention_mask, labels=input_ids).loss
</code></pre>
<p>And the last line cause an issue: AttributeError: ‘Parameter’ object has no attribute ‘_forward_counter’.<br>
The model is GradSampleModule(gpt_neox)</p>
<p>It goes through when I use:<br>
model.disable_hooks(), but I think it is not correct workaround.</p>
Use flexattention with torchrec

Tpopok: <p>Hi Team</p>
<p>Do we have examples to use flexattention with torchrec? I tried by myself but got following errors.</p>
<p>[rank0]: torch._dynamo.exc.BackendCompilerFailed: backend=‘inductor’ raised:<br>
[rank0]: IndexError: tuple index out of range</p>
<p>[rank0]: Set TORCH_LOGS=“+dynamo” and TORCHDYNAMO_VERBOSE=1 for more information</p>
<p>[rank0]: You can suppress this exception and fall back to eager by setting:<br>
[rank0]:     import torch._dynamo<br>
[rank0]:     torch._dynamo.config.suppress_errors = True</p>
<p>Thanks.</p>

Tpopok: <p>Found it is not related with torchrec. please ignore.</p>
GPU Usage graph oscillating

Sourabh_Yadav: <p>My GPUs (2x Tesla V-100) usage loooks like this (attached .png) also is 82 &amp; 79 C temp. normal during training. can I run my gpus like this for 10-12 days.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/4/344dde9d7ba72ce951344fb82a01197791afa133.png" data-download-href="https://discuss.pytorch.org/uploads/default/344dde9d7ba72ce951344fb82a01197791afa133" title="gpuuse"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/4/344dde9d7ba72ce951344fb82a01197791afa133.png" alt="gpuuse" data-base62-sha1="7sHF7GX0qvYO4qlu86mwNWxvUKn" width="496" height="230"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gpuuse</span><span class="informations">496×230 10 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Depending on your use case and workload the graph might be expected. Without any information we can only guess.</p>

Sourabh_Yadav: <p>I am using PeFT on a llama-8B model on tyrone DIT4000TR-48RL.<br>
I have loaded the model in 8-bit to save space.<br>
Also what about temp of 83/79 C of GPUs are they ok?<br>
Also, it would be really helpful if you could suggest some ways to load models efficiently so that I can save space.</p>
How does PyTorch implement batch normalization?

Bjorn_Lindqvist: <p>Sample code to explain what I mean:</p>
<pre><code class="lang-auto">from torch.nn.functional import batch_norm
from torch import from_numpy
import numpy as np

M = np.random.randn(1, 3, 16, 16).astype(np.float32)
m, v, w, b = [np.random.rand(3).astype(np.float32) for _ in range(4)]
M_torch = batch_norm(*[from_numpy(x) for x in [M, m, v, w, b]]).numpy()
m, v, w, b = [p.reshape(1, 3, 1, 1) for p in [m, v, w, b]]
M_np = w * (M - m) / np.sqrt(v + 1e-05) + b
print(np.max(np.abs(M_np - M_torch)))
</code></pre>
<p>The error is quite large. So why isn’t the way I’ve implemented batch normalization using NumPy exactly identical to PyTorch’s method? Note that the epsilon I used is large, 1e-05. But if I use a smaller epsilon like <code>np.finfo(np.float32).eps</code> the results differ even more. So clearly PyTorch is doing something slightly different and I don’t think it can be chalked up to floating point associativity.</p>

ptrblck: <p><a href="https://github.com/ptrblck/pytorch_misc/blob/master/batch_norm_manual.py">This script</a> shows the internal batchnorm approach used in PyTorch.</p>

ptrblck: <p>Did you compare the implementation shared in my post or is this an unhelpful chat bot post?</p>

Bjorn_Lindqvist: <p>Your example does exactly the same calculation as my code. Here is what I get when I print the differences with 10 decimals precision:</p>
<pre><code>print('%12.10f' % (out1 - out2).abs().max())

0.0000002384
0.0000019073
0.0000009537
0.0000019073
0.0000004768
0.0000019073
</code></pre>
<p>So your code also has the same deviations. Why is that? And why is epsilon 1e-5 and not a smaller value?</p>

ptrblck: <p>That’s expected since different orders of mathematical operation cause expected small differences due to the limited floating point precision as seen e.g. here:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(100, 100)
s1 = x.sum()
s2 = x.sum(0).sum(0)

print(s1 - s2)
# tensor(1.5259e-05)
</code></pre>
JetPack 5.0 on Jetson NanoB01?

Bekzod: <p>Can I install JetPack 5.0 on Jetson NanoB01? The reason is that JetPack 4.6.5 is installed on Jetson nano B01 and there are problems using Yolov5 on it.</p>

Bekzod: <p>But I can’t find Pytorch and Torchvision libraries that support CUDA 10.2 in JetPack 4.6.5 on Jetson nano B01. I updated Python version to 3.8. But there is Ultralytics library for Python 3.8 but Pytorch and Torchvision library for Python 3.8 does not support CUDA10.2.</p>

ptrblck: <p>Cross-post from <a href="https://discuss.pytorch.org/t/pytorch-and-torchvision-for-cuda10-2/215865/3">here</a>.</p>
Pytorch and torchvision for CUDA10.2

Bekzod: <p>I have a Jetson nano B01 and JetPack 4.6.6 on it. Are there versions of Pytorch and torchvision that support CUDA 10.2 compatible with JetPack 4.6.6 to use Yolov5ni on Jetson nanoB01?</p>

ptrblck: <p>You can find PyTorch binaries for JetPack 4 <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048">here</a>.</p>

Bekzod: <p>Thank you.But I can’t find Pytorch and Torchvision libraries that support CUDA 10.2 in JetPack 4.6.5 on Jetson nano B01. I updated Python version to 3.8. But there is Ultralytics library for Python 3.8 but Pytorch and Torchvision library for Python 3.8 does not support CUDA10.2.</p>

ptrblck: <p>The link I shared shows PyTorch releases supporting the Jetson platform for JetPack 4. If you need a PyTorch or Yolo version built for another Python version, you might need to build them from source.</p>
Pytorch profiler to profile an inference/ training

dipshi_singh: <ul>
<li>what metrics from PyTorch can be used to break down idle time?</li>
<li>what metrics from PyTorch can be used to analyze HtoD/DtoD/DtoH bandwidth?</li>
<li>what’s the data structure of these metrics?</li>
<li>what’s the precision scale (millisecond?)</li>
</ul>
Licensing for PyTorch Word Usage in Books

orangeeducationpvt: <p>Hi,<br>
I would like to inquire about the licensing terms and conditions related to using PyTorch Word in the context of publishing or referencing it in a book.</p>

ptrblck: <p>Could you describe what “PyTorch Word” is, please?</p>

orangeeducationpvt: <p>We have used this “Mastering Computer Vision with PyTorch 2.0”</p>

ptrblck: <p>I’m not sure if your question is regarding a license or citation, but <a class="mention" href="/u/matt_white">@Matt_White</a> would be the right person to ask.</p>

orangeeducationpvt: <p>Okay, thank you for your information. My question is regarding the license.</p>
<p>Hello <a class="mention" href="/u/matt_white">@Matt_White</a>, could you please advise on whether a license is required to use the term “PyTorch” in a book?</p>

Matt_White: <p>Hi <a class="mention" href="/u/orangeeducationpvt">@orangeeducationpvt</a> , as long as you follow the trademark usage policies (found here: <a href="https://www.linuxfoundation.org/legal/trademark-usage" class="inline-onebox">Trademark Usage | Linux Foundation</a>) you are okay to use the mark.</p>

orangeeducationpvt: <p>Thank you for your prompt response. However, my concern is related to the PyTorch word.</p>

Matt_White: <p>PyTorch is a trademark of the Linux Foundation, and therefore subject to the trademark usage policies that I sent above.</p>
<p>I think what you may be asking is if you can publish a book with the title “Mastering Computer Vision with PyTorch 2.0”, without violating the trademark “PyTorch” since I see you have published that book.</p>
<p>The answer is yes, it is perfectly acceptable to use the word PyTorch in the title of your book, since it is a book about using PyTorch. You do not have to enter into any special licensing agreement to use the word mark “PyTorch” in this context. However you still have to follow the trademark usage policies.</p>
<p>I hope that answers your question.</p>

Matt_White: <p>I would highlight this section of the policy</p>
<p>“There are also some basic rights that everyone has to use any trademark, which are often referred to as “fair use,” and The Linux Foundation does not intend to restrict those rights.  You may make fair use of word marks to make true factual statements.  But fair use does not permit you to state or imply that the owner of a mark produces, endorses, or supports your company, products, or services.  Even when making fair use of a trademark, you should acknowledge the owner of the trademark with a trademark notice, such as the notice displayed on The Linux Foundation project websites.”</p>

orangeeducationpvt: <p>Yes, got your point. Thank you so much for your clarification about the licensing agreement.</p>
CUDA available test is false

Tommy_Zhao: <p>Install pytorch using the following command line<br>
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=10.2 -c pytorch</p>
<p>The result is<br>
pytorch                   1.12.1             py3.10_cpu_0    pytorch<br>
pytorch-mutex             1.0             cpu                     pytorch<br>
It seems that the CPU version is installed</p>
<p>Use the following command to test cuda</p>
<blockquote>
<blockquote>
<blockquote>
<p>print(torch.cuda.is_available())<br>
False</p>
</blockquote>
</blockquote>
</blockquote>
<p>What’s going wrong, any advice welcome!</p>
<p>Attach CUDA related information<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8e8aa2ac325748390947a6b4ad39b341f53a002a.png" data-download-href="https://discuss.pytorch.org/uploads/default/8e8aa2ac325748390947a6b4ad39b341f53a002a" title="CUDA info"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/e/8e8aa2ac325748390947a6b4ad39b341f53a002a.png" alt="CUDA info" data-base62-sha1="kkYPcCxPrDmW7qkNikbF85jdbui" width="690" height="313" data-dominant-color="1D1D1D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">CUDA info</span><span class="informations">1140×518 15.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <aside class="quote no-group" data-username="Tommy_Zhao" data-post="1" data-topic="215928">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tommy_zhao/48/75322_2.png" class="avatar"> Tommy_Zhao:</div>
<blockquote>
<p>It seems that the CPU version is installed</p>
</blockquote>
</aside>
<p>That’s right, as also the version indicates.<br>
You could try to install the older pip wheels if the conda package does not work or you could update to a more recent release.</p>

Tommy_Zhao: <p>Thanks for your suggestion. After I downgraded python and pytorch, I successfully saw True</p>
Attempted to use an uninitialized parameter in <method 'element_size' of 'torch._C._TensorBase' objects>

acmilannesta: <p>I tried to set up this module with lazy linear but still got the error <code>Error: ValueError: Attempted to use an uninitialized parameter in &lt;method 'element_size' of 'torch._C._TensorBase' objects&gt;. This error happens when you are using a </code>LazyModule<code>or explicitly manipulating</code>torch.nn.parameter.UninitializedParameter<code>objects. When using LazyModules Call</code>forward<code> with a dummy batch to initialize the parameters before calling torch functions</code>, thought I already gave a dummy pass?</p>
<pre><code class="lang-auto">class Expert(nn.Module):
    """
    Expert layer for Mixture-of-Experts (MoE) models.

    Attributes:
        l1 (nn.Module): Linear layer for input-to-hidden transformation.
        l2 (nn.Module): Linear layer for hidden-to-output transformation.
        l3 (nn.Module): Additional linear layer for feature transformation.
    """

    def __init__(self, inter_dim: int, device: str = ModelArgs.device):
        """
        Initializes the Expert layer.

        Args:
            dim (int): Input and output dimensionality.
            inter_dim (int): Hidden layer dimensionality.
        """
        super().__init__()  
        self.inter_dim = inter_dim
        self.l1 = nn.LazyLinear(inter_dim, bias=True, device=device)
        self.l3 = nn.LazyLinear(inter_dim, bias=True, device=device)
        self.is_initialized = False

    def forward(self, x: torch.Tensor) -&gt; torch.Tensor:
        """
        Forward pass for the Expert layer.

        Args:
            x (torch.Tensor): Input tensor.

        Returns:
            torch.Tensor: Output tensor after expert computation.
        """
        if not self.is_initialized:
            # Perform a dummy forward pass to initialize LazyLinear layers
            with torch.no_grad():
                _ = self.l1(x)
                _ = self.l3(x)
            input_shape = x.size(1)
            self.l2 = nn.Linear(self.inter_dim, input_shape).to(x.device)
            self.is_initialized = True

        x = F.silu(self.l1(x)) * self.l3(x)
        x = self.l2(x)
        return x

</code></pre>
Adding shape inference to custom operator for ONNX exporting

NimrodR: <p>Hello,</p>
<p>I want to export a PyTorch model to ONNX using <code>torch.onnx.export</code> and I have some custom operators in it. I have managed to add them to TorchScript’s operator registry and I export the model fine to ONNX and Netron shows everything is ok.<br>
The only issue is I am getting this warning:<br>
<code>WARNING: The shape inference of custom_library::custom_op type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.</code><br>
Now I’d love to add some shape inference but I do not now how to. From the relevant docs it seems it needs to be done using the <code>.setType</code> function on the call to <code>g.op</code> in the symbolic function definition. Despite that, <code>.setType</code> receives only types and not dims/shapes.<br>
Furthermore, I couldn’t find any mention of shape inference in Python not in PyTorch’s docs nor in ONNX’s docs. What is the best way to do this then?</p>
<p>Thanks in advance!</p>
<p>NR</p>

NikolayL: <p>Hello <a class="mention" href="/u/nimrodr">@NimrodR</a>!</p>
<p>I’ve experienced the same issue, let me share what I’ve found. Maybe it will be helpful.</p>
<p>Here’s the <span class="hashtag">#73871</span> pytorch issue for documenting shape inference for custom symbolics:<br>
Here’s the <a href="https://github.com/microsoft/onnxruntime/blob/e788cc2a2370946977b91b2456166d466df4d14a/orttraining/orttraining/python/training/ortmodule/_custom_op_symbolic_registry.py#L73" rel="noopener nofollow ugc">example</a> that they are referring to.</p>
<p>But at the same time there is also <span class="hashtag">#81693</span> pytorch issue saying that even with invoking <code>g.op(...).setType()</code> inside the symbolic function it is <em>not sufficient</em> to make shape inference work.</p>
<p>Seems like it’s going to be fixed in next release of torch: <a href="https://github.com/pytorch/pytorch/pull/88622" rel="noopener nofollow ugc">PR</a><br>
I’ve tried latest nightly build torch==1.14.0.dev20221119+cu116, but it’s still not working.<br>
The commit was merged yesterday, possibly it should be in the next build.</p>
<p>BR,<br>
Nikolai</p>

NikolayL: <p>The warning disappeared with .setType and torch==1.14.0.dev20221121+cu116</p>

NimrodR: <p>That’s awesome! I’ll get going with your suggestion, thank you for your thorough research!</p>

markl: <p>Recently ran into this issue. The example linked above refers to the _get_tensor_sizes helper function. When I tried to use this in my custom symbolic function, it returned [None,None,None]. Ideally I’d like to specify that the operator returns something like [A.shape[0], B.shape[1], C.shape[0]]. I’m not sure how to proceed.</p>
Inverse function of `logaddexp`?

jakelevi1996: <p>If I have tensors <code>x</code> and <code>y</code>, and I want to compute <code>z = log(exp(x) + exp(y))</code>, I can use <code>z = torch.logaddexp(x, y)</code>.</p>
<p>Suppose now I have the tensors <code>z</code> and <code>y</code>, and I want to compute <code>x = log(exp(z) - exp(y))</code>.</p>
<p>In other words, I want to compute the inverse function of <code>z = logaddexp(x, y)</code> given <code>y</code>.</p>
<p>Is there a <code>torch</code> function/method that can do this with good numerical stability for large/small numbers?</p>
<p>I tried searching for <code>logsubexp</code> and <code>logdiffexp</code> but nothing came up. One solution would be to do <code>x = z + torch.log1p(-torch.exp(y-z))</code>, but this is not ideal because computing <code>exp(y-z)</code> could overflow/underflow in some cases.</p>

jakelevi1996: <p>Related issue: <a href="https://github.com/pytorch/pytorch/issues/39242" rel="noopener nofollow ugc">“Add numerically stable log1mexp = log(1 - exp(-|x|)) function”</a></p>

jakelevi1996: <p>Also: <a href="https://github.com/pytorch/pytorch/issues/72759" rel="noopener nofollow ugc">“Add softplus inverse”</a></p>

jakelevi1996: <p>After a few experiments with different inputs/dtypes, <code>z + torch.log1p(-torch.exp(y - z))</code> seems to be a reasonably good solution.</p>
<p>If <code>z = logaddexp(x, y)</code>, it’s safe to assume <code>z &gt; y</code>, so (contrary to what I said above) <code>exp(y-z)</code> cannot overflow.</p>
<p>It can underflow if <code>z &gt;&gt; y</code>, but in this case <code>z ~= x</code>, and also <code>z + log1p(-exp(y - z)) ~= z + log1p(-0) == z ~= x</code>, so the answer is still accurate.</p>
<p>For any given precision, as <code>x</code> gets smaller, eventually <code>z == y</code> for all <code>x</code> in floating point representation. So for very small <code>x</code> and <code>z ~= y</code>, the exact inverse cannot be computed exactly anyway. In this case <code>z + log1p(-exp(y - z)) ~= z + log1p(-exp(0)) == z + log1p(-1) == -inf</code>. So I think this solution is as good as can be hoped for.</p>
Anti causal attention mask with key_padding_mask results in nan with MultiHeadAttention

MikePeight: <p>Hello everybody,<br>
I tried to search around for something similar but could not find something similar.<br>
I am having trouble when using an anti causal attention mask as attn_mask(the transpose of the causal one to attend just to the future elements) combined with the key padding mask. I have at the output that the padded embeddings take a nan value. I don’t understand what I am doing wrong. Some suggestions?<br>
I leave the code below:</p>
<pre data-code-wrap="python"><code class="lang-python">from torch.nn import MultiheadAttention
import torch

torch.manual_seed(42)

B, seq_len, embed_dim = 1, 5, 5

mha = MultiheadAttention(embed_dim=embed_dim, num_heads=1, batch_first=True)
x = torch.rand((B, seq_len, embed_dim))
mask = torch.tril(torch.ones(x.size(1), x.size(1)) * float("-inf"), diagonal=-1)
key_padding_mask = torch.tensor([[0, 0, 0, 0, float("-inf")]])
res = mha(x, x, x, attn_mask=mask, key_padding_mask=key_padding_mask)
print(key_padding_mask)
print(mask)
print(res[0])
</code></pre>
<pre><code class="lang-auto">tensor([[0., 0., 0., 0., -inf]])
tensor([[0., 0., 0., 0., 0.],
        [-inf, 0., 0., 0., 0.],
        [-inf, -inf, 0., 0., 0.],
        [-inf, -inf, -inf, 0., 0.],
        [-inf, -inf, -inf, -inf, 0.]])
tensor([[[-0.1386,  0.0392, -0.0841, -0.0103,  0.0435],
         [-0.1620,  0.0555, -0.1101, -0.0131,  0.0673],
         [-0.1162,  0.0409, -0.0969, -0.0043,  0.0651],
         [-0.1792,  0.0674, -0.0597, -0.0299,  0.0199],
         [    nan,     nan,     nan,     nan,     nan]]],
       grad_fn=&lt;TransposeBackward0&gt;)

</code></pre>
Distributed Data Parallel Changing tuple value?

pytorcher: <p>I’m new to DDP and getting some behavior I don’t understand.  I have a line in my code that currently looks like this:</p>
<pre><code class="lang-auto">print(viewTuple)
newTensor = tensor.view(viewTuple)
</code></pre>
<p>viewTuple is defined earlier and always is either [1,-1] or [1,-1,1,1].  This line works properly on single GPU but on multi GPU is giving the error</p>
<blockquote>
<p>RuntimeError: shape ‘[1, 1]’ is invalid for input of size 1000</p>
</blockquote>
<p>This is the case even though the print never shows a [1,1] vector.  Any idea what could be happening? Could the values be being overwritten by the second distributed run?</p>

ptrblck: <p>Is the <code>print</code> statement directly before applying the <code>view</code> operation showing <code>[1, -1]</code>, but then fails with in the <code>view</code> operation? If so, then I haven’t seen such a behavior before of Python <code>list</code>s being manipulated silently by using multi-processing.</p>

pytorcher: <p>Yes, the <code>print</code> is directly before the <code>view</code>.  It prints a bunch of vectors with a -1 in them to allow for reshaping but then I get the error with a [1,1] array.  I also tried adding a <code>print("flush", flush=True)</code> after as well to make sure all the prints are being flushed.  Any other thoughts or is this something that I may need to create a minimal example to reproduce for debugging?</p>
What is the right way to structure `input` and `label` while fine-tuning decoder only model

Seungjun_Lee: <p>I am working on fine-tuning a Llama(decoder only) model on SQuAD Dataset, and wondering what is the right way</p>
<p>to structure inputs and labels.</p>
<p>Option1:</p>
<p>input: “SOS context:{context} question:{question} answer: pad …. pad”</p>
<p>label: “{answer} EOS pad pad … pad”</p>
<p>Option2:</p>
<p>input: “SOS context:{context} question:{question} answer: pad …. pad”</p>
<p>label: “pad … pad {answer} EOS pad … pad”</p>
<p>Option3:</p>
<p>input: “SOS context:{context} question:{question} answer: {answer} pad …. pad”</p>
<p>label: “context:{context} question:{question} answer:{answer} EOS pad …. pad”</p>
Why is torch.compile so fast?

hello_e: <p>hello, my question is besides operator fusion and CUDA graph, does it use any other technologies, such as memory management similar to vllm? Also, is there a difference between the CUDA graph used by torch.compile and replaying with torch.cuda.graph?</p>
<p>I tested my handwritten operator fusion and found it to be two to three times faster than the fusion done by torch.compile. Additionally, I incorporated CUDA graph for replay, yet the end-to-end inference speed is only 10% faster than torch.compile. Is there anywhere else I might have overlooked?</p>

ptrblck: <p>Are you sure you are profiling the code correctly? I.e. did you compare timelines of e.g. <code>nsys</code> profiles or used synchronized host timers for the comparison?</p>
Question about gradient calculation in backward() of actor network of DDPG

jayjay: <p>Hi,<br>
My question is about gradient calculation and computational graphs in DDPG.</p>
<p>Following is the code snippet for actor-critic update in DDPG.</p>
<p>self.actor_optimizer = torch.optim.Adam(self.actor.parameters(), lr=actor_lr, weight_decay=actor_decay)<br>
self.critic_optimizer = torch.optim.Adam(self.critic.parameters(), lr=critic_lr, weight_decay=critic_decay)</p>
<p><strong>Sample from the experience replay buffer</strong><br>
state, action, next_state, reward, not_done = replay_buffer.sample(batch_size)</p>
<p><strong>Compute the target Q-value</strong><br>
** The computational graph with target_Q is created**<br>
target_Q = self.critic_target(next_state, self.actor_target(next_state))<br>
<strong>make a new tensor for target_Q.detach()</strong><br>
target_Q = reward + (not_done * self.discount * target_Q).detach()</p>
<p><strong>Get the current Q-value estimate</strong><br>
<strong>The computational graph with self.critic is created</strong><br>
current_Q = self.critic(state, action)</p>
<p><strong>Compute the critic loss</strong><br>
critic_loss = F.mse_loss(current_Q, target_Q)</p>
<p><strong>Optimize the critic</strong><br>
self.critic_optimizer.zero_grad()<br>
<strong>gradients w.r.t. weights of self.critic is calculated and the computational graph is freed</strong><br>
critic_loss.backward()<br>
<strong>updates weights of self.critic based on gradients</strong><br>
self.critic_optimizer.step()<br>
<strong>Compute the actor loss</strong><br>
<strong>the computational graph with self.critic and self.actor is created</strong><br>
actor_loss = -self.critic(state, self.actor(state)).mean()</p>
<p><strong>Optimize the actor</strong><br>
self.actor_optimizer.zero_grad()<br>
actor_loss.backward()<br>
self.actor_optimizer.step()</p>
<p>My question is “actor_loss.backward()” calculates gradients w.r.t weights of both self.critic and self.actor and if so, gradients of self.critic calculated in critic_loss.backward() remains so it seems accumulated. However, it doesn’t affect updating self.actor because self.actor_optimizer only cares about self.actor’s parameters. Do I understand correctly?</p>
<p>My second question is that the computational graph for Target_Q looks not freed. correct? if so, how to free the computational graph for Target_Q without backward() operation?</p>
combining TEXT.build_vocab with BERT Embedding

Muhammad_Alfian: <h2><a name="p-464128-questions-and-help-1" class="anchor" href="#p-464128-questions-and-help-1"></a>Questions and Help</h2>
<p><strong>Description</strong></p>
<p>Hi, we can use glove embedding when building vocab, using<br>
something like:</p>
<pre><code class="lang-auto">MIN_FREQ = 2

TEXT.build_vocab(train_data, 
                 min_freq = MIN_FREQ,
                 vectors = "glove.6B.300d",
                 unk_init = torch.Tensor.normal_)
</code></pre>
<p>However, I want to use BERT embedding because I need a sophisticated model to compare the performance of multiple embeddings. How can I use BERT in build_vocab?</p>
How to save torch.compile so we don't need to re-compile

vgoklani: <p>Is it possible to save the torch.compile <code>state</code> so we don’t need to re-compile each time when running the same exact model? It takes over <em>15</em> minutes to compile, and it’s presumably doing the same exact work each time. We are running LLama-3 70B via tensor-parallel with int8 quantization.</p>

qpwo: <p>Do you have this in bashrc?</p>
<pre><code class="lang-auto">export TORCHINDUCTOR_FX_GRAPH_CACHE=1
export TORCHINDUCTOR_CACHE_DIR=/home/ubuntu/.inductor_cache
</code></pre>
Creating a wrapper for layers and tensors

J0hn: <p>Hello, I need some advice since I am stuck on this.</p>
<p>I need to change the background arithmetic that happens for some reason. My goal is to create my own custom operators.  so that everything in the backend will be using my own arithmetic implementation. It has worked for keras but it was simple. My logic is to create a wrapper class for the torch tensor and the torch layers. The reason I want to wrap the layers is because I want it to look the same for the users. Before I start hammering at this any advice??</p>
AOT Inductor compiled kernel format

etaf: <p>We would like to know whether the current GPU AOT (Ahead-of-Time) Inductor is suitable for compiling kernels on the server and then deploying them to edge devices under the following conditions, and what performance impacts might arise:</p>
<ol>
<li>The driver on the server is different from the driver on the deployment device.</li>
<li>The GPU model on the server is different from the GPU model on the deployment device.</li>
</ol>
<p>We understand that for CUDA, the AOTI-compiled kernel is in the <code>.cubin</code> format, which is an architecture-specific native binary. The same <code>.cubin</code> file is not guaranteed to be compatible across different architectures.</p>
<p>For XPU, the AOTI-compiled kernel is in the SPIR-V format, which is an intermediate representation (IR) that can be further compiled on the deployment device into a binary tailored to the current driver and GPU architecture. However, this introduces additional compilation time on the deployment side. Of course, we can cache the compiled binary to avoid repeated compilation.</p>
<p>Should we consider the situation that different gpu/dirver between compiler end and deploy end for AOTI?<br>
Given the two compiled kernel formats described above, we would like to hear your thoughts on which approach is better.</p>
Low GPU Utilization without obvious bottlnecks

Rezzy139: <p>I am running training on a rather simple deep learning network (~800kb).<br>
LSTM with 2 layers and 128 neurons<br>
Feed Forward with 2 linear layers and 64 neurons<br>
Final Feed forward with 2 linear layers and 64 neurons</p>
<p>However, I am getting low GPU utilization at around 45% &amp; 140W when looking at nvidia-smi on my RTX 6000</p>
<p>I am not sure why this is the case as there should not be bottlenecks as data is all loaded into memory, and I sent the tensors straight to the GPU. So the GPU is doing the shuffling and slicing on batch sizes (previously I did this on the CPU, but the CPU was maxed out so it was the bottleneck).</p>
<p>With 2 million inputs (~1000 features each input), it uses about 18GB of GPU Memory out of 24GB on my RTX 6000</p>
<p>I am using a batch size of 1024 and its taking about 60 seconds an epoch. It seems training speed is linear with batch size as with 512 it was about 120seconds, and 2048 was 33 seconds.<br>
My forward/backward pass is the standard pytorch implementation.</p>
<p>When I added in torch.cuda.amp.autocast/GradScaler.and torch.backends.cudnn.benchmark = True, my utilization and wattage when down but time per epoch stayed the same.</p>
<p>I am wondering if you can help me troubleshoot as I need to train this model as fast as possible. I will also run into the issue down the line of having too much data to fit into the GPU (but will fit in RAM), so any help on an efficient solution there would also be appreciated. I previously used DataLoaders/Dataset but it was really slow, even slower with pinned_memory and num_workers &gt; 0.</p>
<p>Thanks! Let me know if any further information is required</p>
Call order of hooks

sternj: <p>If there are <code>pre_forward</code> and <code>forward</code> hooks set on two different instances of subclasses of <code>torch.nn.module.Module</code> from the main thread, is there any circumstances in which the following order of calling could occur?</p>
<pre><code class="lang-auto">mod1_pre_forward -&gt; mod2_pre_forward -&gt; mod1_forward -&gt; mod2_forward
</code></pre>
<p>That is to say-- if the forward pre-hook of module 1 is called before the forward hook of module 2 and the forward post-hook of module 1 has not yet been called, is there any circumstance in which the forward post-hook of module 1 will be called before the forward post-hook of module 2?</p>
Loss.backward() causing unexpected errors

gabe_j: <p>I am trying to train a BERT model with 3 output heads on 3 tasks. When I begin the training I run into the issue ‘RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn’. I have set most of the parameters to requires_grad=False because for the first epoch I only want to update the output layers. This approach has worked well with 2 tasks/heads, and when all I change is adding the third head/task I get this issue. Nothing else is being changed.<br>
Here is where the error occurs:</p>
<pre><code class="lang-auto">def training_step(self, model: torch.nn.Module, inputs: Dict[str, Union[torch.Tensor, Any]]) -&gt; List[torch.Tensor]:
        model.train()
        if self.l0_module is not None:
            self.l0_module.train()
        inputs = self._prepare_inputs(inputs)

        distill_loss = None
        distill_ce_loss = None
        # if self.teacher_model is not None:
        with torch.no_grad():
            # only retain inputs of certain keys
            teacher_inputs_keys = ["input_ids", "attention_mask", "token_type_ids", "position_ids", "labels",
                                    "output_attentions", "output_hidden_states", "return_dict"]
            teacher_inputs = {key: inputs[key]
                                for key in teacher_inputs_keys if key in inputs}
            self.shortens_inputs(teacher_inputs)
            teacher_outputs = self.teacher_model(**teacher_inputs)
        self.shortens_inputs(inputs)
        student_outputs = model(**inputs) #! get the two outputs

        zs = {key: inputs[key] for key in inputs if "_z" in key} #! extract the zs
        distill_loss, distill_ce_loss, loss = self.calculate_distillation_loss(
            teacher_outputs, student_outputs, zs)

        lagrangian_loss = None
    
        loss.backward()
</code></pre>
<p>Here is the model init:</p>
<pre><code class="lang-auto">def __init__(self, config):
        super().__init__(config)
        self.config = config
        self.bert = CoFiBertModel(config)

        self.do_layer_distill = getattr(config, "do_layer_distill", False)
        self.num_labels = 2
        self.task = None

        self.dropout = nn.Dropout(p=0.1, inplace=False)

        self.task1_classifier = nn.Linear(config.hidden_size, 2)
        self.task2_classifier = nn.Linear(config.hidden_size, 2)
        self.task3_classifier = nn.Linear(config.hidden_size, 2)

        self.post_init()
        if self.do_layer_distill:
            self.layer_transformation = nn.Linear(
                config.hidden_size, config.hidden_size)
        else:
            self.layer_transformation = None
</code></pre>