akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch.divide only where denominator is non-zero

abhisek: <p>In <code>numpy</code> I can do the following to avoid division by zero:</p>
<pre><code class="lang-python">a = np.random.randint(0, 10, 100)
b = np.random.randint(0, 10, 100)
c = np.zeros_like(a, dtype=np.float32) # It can be anything other than zero
c = np.divide(a, b, out=c, where=(b!=0))
</code></pre>
<p>In <code>torch.divide</code> there is <strong>no</strong> <code>where</code> argument for masking. Only way seems to be replacing <code>inf</code> with desired value after the division takes place. Is there any better way to do this in <code>pytorch</code>?</p>

ptrblck: <p>Would selecting the desired values in <code>a</code> and <code>b</code> before applying the division work?<br>
E.g. you could create a mask first and use it to index both tensors where the condition would be <code>b!=0</code>.</p>

abhisek: <p>Yes, indeed your(<a class="mention" href="/u/ptrblck">@ptrblck</a>) solution works pretty well. For anyone who’s looking for solution using torch see the snippet below:</p>
<pre><code class="lang-python">import torch

# numerator: tensor([2., 2., 0., 5., 7., 3., 4., 3., 6., 5.])
a = torch.randint(0, 10, (10,), dtype=torch.float32)

# denominator: tensor([3., 3., 0., 4., 5., 4., 7., 8., 0., 4.])
b = torch.randint(0, 10, (10,), dtype=torch.float32)

# initialize output tensor with desired value
c = torch.full_like(a, fill_value=float('nan'))

# zero mask
mask = (b != 0)

# finally perform division
c[mask] = a[mask] / b[mask]

# output: tensor([0.6667, 0.6667,    nan, 1.2500, 1.4000, 0.7500, 0.5714, 0.3750,    nan, 1.2500])

</code></pre>

jakelevi1996: <p>Another solution is to use <code>where</code> before you divide (this avoids initialising <code>c</code> and then conditionally modifying its elements in-place - I think this looks more elegant, but I haven’t measured its efficiency):</p>
<pre data-code-wrap="python"><code class="lang-python">good_inds = (b != 0)
fill_value = 42 # or whatever
c = (
    torch.where(good_inds, a, fill_value) /
    torch.where(good_inds, b, 1)
)
</code></pre>
Windows Debug Build GIL Error When Importing Torch

ikalinic: <p>Hi everyone. Does anyone know the correct way of building PyTorch in Debug mode on Windows? I’ve been able to build using <code>set DEBUG=1</code> and running <code>python setup.py develop</code> with also setting the <code>LIB</code> env variable to the Python libs, but this causes the following error when running <code>import torch</code> with Python:</p>
<p>Fatal Python error: _PyInterpreterState_GET: the function must be called with the GIL held, after Python initialization and before Python finalization, but the GIL is released (the current Python thread state is NULL)</p>
<p>The error above happens with no stack trace.</p>
<p>After running <code>import torch</code> with <code>python_d.exe</code> the GIL error is not present, however I get an ImportError related to numpy (attached screenshot). This is probably due to some of the dependencies (like numpy) being in Release mode, because they were just installed with <code>pip install</code>.Have I done something wrong? Is there any way to build PyTorch in full Debug mode on Windows, rather than in Release with Debug symbols (<a href="https://discuss.pytorch.org/t/fail-to-build-debug-version-on-windows/81127" class="inline-onebox">Fail to build debug version on windows</a>, <a href="https://discuss.pytorch.org/t/fail-to-import-torch-on-windows-debug-build/81347" class="inline-onebox">Fail to "import torch" on Windows debug build</a>)?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/8/58506a5d14f3b120e566a3f7a5dbff31bc334bdf.png" data-download-href="https://discuss.pytorch.org/uploads/default/58506a5d14f3b120e566a3f7a5dbff31bc334bdf" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/58506a5d14f3b120e566a3f7a5dbff31bc334bdf_2_453x500.png" alt="image" data-base62-sha1="cBgj1fvXPmxZNCFMy6Ty4XNK2aH" width="453" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/58506a5d14f3b120e566a3f7a5dbff31bc334bdf_2_453x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/8/58506a5d14f3b120e566a3f7a5dbff31bc334bdf_2_679x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/8/58506a5d14f3b120e566a3f7a5dbff31bc334bdf.png 2x" data-dominant-color="171717"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">820×905 315 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I tried following the suggestions from the error without success.<br>
These errors happen in plain system Python installation, venv and conda env.</p>

charlesxsh: <p>In Windows debug version of python (python_d), it requires C extension also in debug version. Which means, you should build numpy in debug version to be used with python_d</p>
Experiences using an RTX 5080 with Resnet50 / Efficientnet_b3 / Inception_v3?

emerth: <p>I’m looking for some info to compare to my own observations of an RTX 5080 I bought a couple weeks ago.</p>
<p>The architectures noted in the title and batch size 16 or 32.</p>
<p>What does ‘nvidia-smi dmon’ show for power, SM% for you?</p>
<p>–</p>
<p>My 5080 card is drawing between 225W and 270W depending on the architecture being trained, and shader utilization is higher than 95%. It is ~1.7x the speed of a 3080/12G in terms of time to process an epoch’s worth of files.</p>
<p>In the past, using RTX 3080 or A6000 or RTX 2080ti, the card would ramp up to near max power draw. This 5080is rather demure by comparison.</p>
<p>Imagine it!  An RTX card that does not want to eat the world! It’s great.</p>
<p>Training ResNet50, Efficientnet_b3, Inception_v3 on Imagenet, 224x224 images, batch size 16 or 32.</p>
<p>Thanks!</p>

python: <p>Hi Emerth,</p>
<p>Thanks for sharing your experience with the RTX 5080! It’s interesting to see how it compares to previous models. Your power draw numbers are pretty consistent with what I’ve observed, too. I’ve seen similar ranges while training with ResNet50 and EfficientNet, and like you mentioned, the shader utilization is impressively high.</p>
<p>I also noticed that the 5080 doesn’t ramp up power as aggressively as the previous cards, which is refreshing. It seems to balance performance and power efficiency quite well.</p>

emerth: <p>It makes me wonder what would be possible with a similar product that did not contain hardware for ROPs, ray tracing support, image scaling support. Just all cuda shaders.</p>
How to obtain the values of query and key in the attention module"

wangzhexuan95: <p>How to obtain the values of <code>query</code> and <code>key</code> in the attention module"</p>

ajayrkumar: <p>I believe you are talking about <code>torch.nn.MultiheadAttention</code>.</p>
<p>Multihead attention expects you to pass the <code>query</code>, <code>key</code> and <code>value</code> tensors in its forward pass. So, you would have access to them already.</p>
<p>I would be able to help you more if you could explain your question in a bit more detail.</p>
Where to get the version string of Lightning runs?

evilroach: <p>I’m using Lightning to train my model.</p>
<p>Lightning has a nice feature i.e. every time it runs, it creates a sub-dir in <code>./lightning_logs</code>, and nams them like <code>version_0</code>, <code>version_1</code>, <code>version_2</code>…</p>
<p>I want to utilize these strings to identify my prediction result sets when saving them into DB, but I don’t know How to get these version string( <strong>Not the release version of Lightning</strong> ) of Lightning runnings?</p>
<p>Thanks!</p>

ptrblck: <p>This might be a great question for the <a href="https://github.com/Lightning-AI/pytorch-lightning/discussions">Lightning forum</a>. Based on <a href="https://github.com/Lightning-AI/pytorch-lightning/discussions/7897">this question</a> it seems you could change or get the version from the logger.</p>

evilroach: <p>Thanks, ptrblck!</p>
<p>I don’t know there is a dedicated Lightning forum before.</p>

ptrblck: <p>No worries! I just wanted to point towards it in case the quick search did not help. Was the linked issue helpful?</p>

evilroach: <p>Yeah, the link you given inspired me to finger out the version-string, even although a Logger obj has no direct attribute offering that.</p>
<p>We have to manually extract Ver-Str from Attribute <code>log_dir</code> of a Logger.</p>
<p>Thank you again!</p>
We are not getting gradients after custom loss creation

Filippo_Portera: <p>We are playing with MNIST and we created an algorithm that uses splines to do inference.<br>
The problem is that whenever we want to get the gradients of the “weights” of the loss function we obtain None values. The “weights” are initialized with require_grad=True .<br>
Here’s a piece of code:<br>
def training():<br>
global alpha, trs</p>
<pre><code>       for i in range(pos * bs, (pos + 1) * bs):
             ...
        
             pred = torch.tensor([torch.tensor([torch.dot(alpha[q], v[q]) for q in range(0, 1)])])
        
             pred.sum().backward()

             print(alpha.grad)
</code></pre>
<p>Here alpha.grad is None. The weights are the alpha’s variables and matXtr[i][j] is the input matrix X.<br>
Again, we don’t understand why gradients are None.</p>

ptrblck: <p>Creating a new tensor will detach it from the computation graph, so use differentiable operations such as <code>torch.stack</code> or <code>torch.cat</code>.</p>
How to test new native function

Shane_Stevenson: <p>Hello, I am following the autograd onboarding lab (<a href="https://github.com/pytorch/pytorch/wiki/Autograd-Onboarding-Lab" class="inline-onebox" rel="noopener nofollow ugc">Autograd Onboarding Lab · pytorch/pytorch Wiki · GitHub</a>), and on step 3 it instructs “You can then add an OpInfo entry (in <a href="https://github.com/pytorch/pytorch/blob/master/torch/testing/_internal/common_methods_invocations.py" rel="noopener nofollow ugc">common_methods_invocations.py</a>) to be able to test the new operator.”</p>
<p>Is there any documentation on this common_methods_invocations.py file? What is this file’s purpose and how might I learn how to add to it? Furthermore, how will adding to this file allow me to test my new native function?</p>

Shane_Stevenson: <p>I solved my problem. A very descriptive comment in “core.py” describes what OpInfos are and how to write them.</p>
Signal convolution in torch

Francesca_DeBenetti: <p>Hello!<br>
I am convolving two 1D signals with scipy.signal.fftconvolve: <code>c = fftconvolve(b, a, "full")</code>. I would like to replace the fftconvolve function with a torch function. I tried to use torch.nn.functional.conv1d, but it doesn’t return the result I expected. My signals have the same length (and not start/end with 0).</p>
<p>For the sake of completeness, I tested the following code:</p>
<pre><code class="lang-auto">from scipy import signal
from torch.nn.functional import conv1d
a = np.repeat([0., 1., 0.], 100)
b = np.repeat([0., 1., 0.], 100)
c = signal.convolve(a, b, mode='full')
c_fft = signal.fftconvolve(a, b, mode='full')

torch_a = torch.Tensor(a)
torch_a = torch.unsqueeze(torch_a, 0)
torch_a = torch.unsqueeze(torch_a, 0)
torch_b = torch.Tensor(b)
torch_b = torch.unsqueeze(torch_b, 0)
torch_b = torch.unsqueeze(torch_b, 0)
torch_c = conv1d(torch_a, torch_b, padding=len(a))
</code></pre>
<p>Here conv1d behaves as expected, but if I set <code>a = b = np.repeat([1., 1., 0.], 100)</code> the result is no longer the same (I assume because the signal doesn’t start/end with 0).</p>
<p>Any help is welcome!</p>
<p>Best,<br>
Francesca</p>

mvalente: <aside class="quote no-group" data-username="Francesca_DeBenetti" data-post="1" data-topic="165175">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/francesca_debenetti/48/51515_2.png" class="avatar"> Francesca_DeBenetti:</div>
<blockquote>
<pre><code class="lang-auto">torch_c = conv1d(torch_a, torch_b, padding=len(a))
</code></pre>
</blockquote>
</aside>
<p>try reducing the padding</p>
<pre><code class="lang-auto">torch_c = conv1d(torch_a, torch_b, padding=len(a)-1)
</code></pre>

Francesca_DeBenetti: <p>No improvement, unfortunately <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>

lsrosa: <p>2 years later, but I was looking for the same thing.<br>
I found this <a href="https://stackoverflow.com/questions/78421194/1d-signal-convolution-using-pytorch" rel="noopener nofollow ugc">StackOverflow</a> solution which did the trick for me</p>
Autocast behaviour in different GPUs?

strivn: <p>Hi,</p>
<p>I’m doing my deep learning assignment and I moved over my code from Kaggle with P100 to a computing cluster with V100. The code and config stays the same, but when I ran the training, the model can’t converge (loss goes up, loss goes to nan, etc) even though it’s doing just fine in Kaggle.</p>
<p>Any ideas on what could’ve happened or how to fix this?</p>
<p>I tried isolating the issue, it may be due to torch.autocast.<br>
When I removed with <code>torch.autocast(device_type='cuda')</code>, it works just fine.</p>
<pre><code class="lang-auto">def train_model(model, train_loader, criterion, optimizer):

    model.train()
    batch_bar = tqdm(total=len(train_loader), dynamic_ncols=True, leave=False, position=0, desc='Train')

    total_loss = 0

    for i, data in enumerate(train_loader):
        optimizer.zero_grad()

        x, y, lx, ly = data
        x, y = x.to(device), y.to(device)
        lx, ly = lx.to(device), ly.to(device)

        with torch.autocast(device_type='cuda'):
            h, lh = model(x, lx)
            h = torch.permute(h, (1, 0, 2))
            loss = criterion(h, y, lh, ly)

        total_loss += loss.item()

        batch_bar.set_postfix(
            loss="{:.04f}".format(float(total_loss / (i + 1))),
            lr="{:.06f}".format(float(optimizer.param_groups[0]['lr'])))

        batch_bar.update() 
        
        scaler.scale(loss).backward() 
        scaler.step(optimizer) 
        scaler.update() 


        if scheduler:
            if isinstance(scheduler, (torch.optim.lr_scheduler.CosineAnnealingLR, 
                                      torch.optim.lr_scheduler.OneCycleLR)):
                scheduler.step()
        
        del x, y, lx, ly, h, lh, loss
        torch.cuda.empty_cache()

    batch_bar.close()

    return total_loss / len(train_loader)
</code></pre>
<p>Best,</p>

deepsthecoder: <p>Hey Ivan,<br>
Nan in loss is occurring due to gradient underflow issues that happens in V100 GPU, as fp16 values are supported in V100s and not in Kaggle ones. I assume in Kaggle your model is falling back to fp32 as there could not be complete support for fp16.</p>
<p>You could try gradient scaling for the Nan issue and loss stabilization.</p>
GroupNormalizer gives odd results in pytorch-forecasting

Eyzzle: <p>Hello everyone,</p>
<p>I am currently playing with some dummy data to perform deep learning tasks. I was checking the normalization of my data, which seems wrong.<br>
Here is how my data is defined:</p>
<pre><code class="lang-auto">num_rows = 100
df = pd.DataFrame({
    "time": range(num_rows),
    "temperature": [100*(-1)**n for n in range(num_rows)],
    "group": [1]*num_rows
})
</code></pre>
<p>Clearly, the temperature has a <strong>mean of 0</strong> and <strong>a standard deviation (scale) of 100</strong>, however if I run the following</p>
<pre><code class="lang-auto">gn = GroupNormalizer(groups=["group"])
gn.fit(df["temperature"], df)
print(gn.norm_)
</code></pre>
<p>I end up with the following results:</p>
<pre><code class="lang-auto">       center       scale
group                    
1         0.0  100.504758
</code></pre>
<p>That’s why I was wondering what is going on.<br>
Is the error coming from numerical approximations ? (but the error seems large in comparison of the number of rows)<br>
Is it coming from the way GroupNormalizer computes the normalization ?<br>
Am I missing something ?</p>
<p>Thanks for reading!</p>
What's the difference between .scalar_type vs .dtype?

rdgq: <p>What is the difference between these two methods on tensors?</p>

ptrblck: <p>There should be no difference as <code>ScalarType</code> is the C++ side for <code>dtype</code>.</p>

rdgq: <p>The reason I ask is because I notice that they have different return types:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38d1c11e2643b7f7f7c31c4a1e9075b0f4271ca4_2_690x35.png" alt="Screenshot 2025-05-22 at 9.12.02 PM" data-base62-sha1="86E8mWPTSkXMtmf9CJXstfP5BaY" width="690" height="35" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38d1c11e2643b7f7f7c31c4a1e9075b0f4271ca4_2_690x35.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38d1c11e2643b7f7f7c31c4a1e9075b0f4271ca4_2_1035x52.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/8/38d1c11e2643b7f7f7c31c4a1e9075b0f4271ca4_2_1380x70.png 2x" data-dominant-color="212121"></p>
<p>I see they are convertible to each other, should we just always prefer scalar_type then?</p>
What is the "persistent algorithm" in GRU and LSTM?

turian: <p><a href="https://pytorch.org/docs/stable/generated/torch.nn.LSTM.html" rel="noopener nofollow ugc">LSTM</a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.GRU.html" rel="noopener nofollow ugc">GRU</a> docs have the following notes:</p>
<p>“If the following conditions are satisfied: 1) cudnn is enabled, 2) input data is on the GPU 3) input data has dtype <code>torch.float16</code> 4) V100 GPU is used, 5) input data is not in <code>PackedSequence</code> format persistent algorithm can be selected to improve performance.”</p>
<p>What is persistent algorithm and how can I select it?</p>

ptrblck: <p>You cannot select it manually and it will automatically be used, if the specified conditions are met.<br>
<a href="https://on-demand.gputechconf.com/gtc/2017/presentation/s7122-stephen-jones-cuda-optimization-tips-tricks-and-techniques.pdf">This GTC talk</a> gives some information on persistent kernels, which basically try to avoid memory “movement” and try to reuse values once they are loaded.</p>

aasdls: <p>Sorry to re-open this thread, but wanted to ask some questions to check if someone figured it out already.</p>
<p>I was testing LSTMs and GRUs on a A100 and on a GB200. The A100 seems to use the persistent algorithm for some combinations of batch sizes and hidden dim sizes; the GB200 doesn’t seem to do it.</p>
<p>Apart from that, it’s not clear to me what are the rules triggering the usage of the persistent algorithm. e.g. when training both on a A100, a unidirectional LSTM with sequence lenght 2048 and batch size 16 uses the persistent kernel, whereas a bidirectional LSTM with sequence lenght 2048 and batch size 8 does not.</p>
<p>Does anyone know the boundaries at which the persistent algorithm will be chosen instead of the standard one?</p>
shape '[64, 32, 3, 3]' is invalid for input of size 1020

Gaby: <p>Hi ,</p>
<p>I need your help<br>
this is my output when i run this</p>
<p>wwwdata@serveur:~/downloads/libtorch-yolov3$ ./yolo-app  ../imgs/person.jpg</p>
<p>output :</p>
<p>terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  shape ‘[64, 32, 3, 3]’ is invalid for input of size 1020<br>
Aborted (core dumped)</p>
<p>this is my environment:</p>
<p>wwwdata@serveur:~/downloads/libtorch-yolov3$ lsb_release -a<br>
No LSB modules are available.<br>
Distributor ID: Ubuntu<br>
Description:    Ubuntu 22.04.5 LTS<br>
Release:        22.04<br>
Codename:       jammy</p>
<p>wwwdata@serveur:~/downloads/libtorch-yolov3$ hostnamectl<br>
Static hostname: serveur<br>
Icon name: computer-vm<br>
Chassis: vm<br>
Machine ID: dd0762ee40c5444682b0807ec155ec6f<br>
Boot ID: 7bcfc3a214134b92970d919c4f3a2757<br>
Virtualization: oracle<br>
Operating System: Ubuntu 22.04.5 LTS<br>
Kernel: Linux 5.15.0-139-generic<br>
Architecture: x86-64<br>
Hardware Vendor: innotek GmbH<br>
Hardware Model: VirtualBox</p>
<p>Regards</p>

ptrblck: <p>Somewhere in your code you are trying to <code>reshape</code> or <code>view</code> a tensor with an invalid shape:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(1020)
x.view(64, 32, 3, 3)
# RuntimeError: shape '[64, 32, 3, 3]' is invalid for input of size 1020
</code></pre>
<p>and would need to narrow down the failing line of code.</p>
Lowering exir to quantized operations (no delegate)

jpiat: <p>Hi,</p>
<p>i’m working with executorch and torch.ao to enable quantization aware training in our workflow and possibly lowering a quantized implementation using executorch but no delegate. I wrote a Quantizer that can be configured to match multiple qscheme and i get a properly annotated graph than can be used in QAT. After pt2e conversion i get a model with inserted quantizer/dequantizer that uses the ATen operators (floating points) that i can convert to edge. PyTorch has a set of quantized ATen operators (<a href="https://github.com/pytorch/pytorch/tree/main/aten/src/ATen/native/quantized" class="inline-onebox" rel="noopener nofollow ugc">pytorch/aten/src/ATen/native/quantized at main · pytorch/pytorch · GitHub</a>) and i’d like to be able to use them to run the model inference. Looking through executorch code, there is a mention of <code>replace_quantized_partition_with_op</code> in exir/backend/utils but i don’t see any use of it in the repository. Is there any way to replace dq/op/q partitions in the exir graph to use exir/ATen quantized operators without using a delegate ? In <code>executorch/kernels/aten/functions.yaml</code> there is no ATen quantized operators registered and the quantized folder only registers a very limited set of operations (mainly quantize/dequantizers), is there any plan to have quantized operators supported in the runtime with no delegate ?</p>

swolchok: <aside class="quote no-group" data-username="jpiat" data-post="1" data-topic="216772">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jpiat/48/75700_2.png" class="avatar"> jpiat:</div>
<blockquote>
<p>In <code>executorch/kernels/aten/functions.yaml</code> there is no ATen quantized operators registered</p>
</blockquote>
</aside>
<p>I believe ATen mode is not supported in OSS (and it’s not supported on mobile/embedded devices anyway AFAIK), so I don’t think this would help you even if it existed.</p>
<blockquote>
<p>no delegate</p>
</blockquote>
<p>Can you say more about why XNNPACK is not suitable for your use case?</p>
<blockquote>
<p>is there any plan to have quantized operators supported in the runtime with no delegate ?</p>
</blockquote>
<p><a class="mention" href="/u/mergennachin">@mergennachin</a> / <a class="mention" href="/u/manuelcandales">@manuelcandales</a> do either of you know if this is in the plan?</p>

jiaoliang_yu: <p>Hi,<br>
I’m attempting to replace the gemm with a custom acceleration library. However, I’ve executed llama3_2_3b_spinquant pte and found that it uses a delegate to call the xnnpack operators. There are too many kernels in xnnpack, making it difficult for me to identify the specific kernel. Can I use native operators to perform computations on quantized operators instead of using the delegate, or is it possible to find the corresponding xnnpack kernel using the profiling information provided below?<br>
│ Execute │ Fully Connected (NC, QD8, F32, QB4W) GEMM <span class="hashtag-raw">#1</span> │</p>
Suggestions on how to use the Torch Profiler API

mEm: <p>Hi everyone, I’d like some suggestions from experts on how to use Torch profiler API from the C++ frontend.<br>
I was reading through the code, and I guess the core in in <code>/torch/csrc/profiler/</code>.<br>
However, I expected a Profiler class to be provided, while it does not seem to be the case.<br>
I suppose in an application one should include <code>/torch/csrc/profiler/api.h</code>, which only includes <code>observer.h</code> in which only some config stuff is declared/defined.</p>
<p>Also, it seems that profiler related stuff is present in two different say Torch modules: (1) <code>torch::profiler</code> and <code>torch::autograd::profiler</code>. I suppose the former is to be used, but I might be wrong. However, for example for the Kineto client, it looks like that the interface implementation in <code>torch/csrc/profiler/</code> is a wrapper API around the implementation found under <code>/csrc/autograd</code>.</p>
<p>So, I am a bit confused on how the Profiling infrastructure is setup in Torch.<br>
I’d love some guidance/help on how to use this functionality <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>PS: I was expecting to see a backend-like structure as it is done for the distributed framework, where there is the base <code>Backend</code> virtual class, and all its implementations depending on the compile options. So something like:</p>
<pre><code class="lang-auto">class Profiler {}
class KinetoProfiler : public Profiler {}
</code></pre>
Fail to run torch model on Vulkan backend

lzhao4ever: <p>I compiled torch with vulkan backend, and saved the model with vulkan backend. However, when I do the inference on android device, it throwed the following error:<br>
com.facebook.jni.CppException: Could not run ‘vulkan_prepack::run_conv2d_context’ with arguments from the ‘CPU’ backend.</p>
<p>Please let me know what’s the cause of this issue. Thanks</p>

python: <p>That error usually means the model or one of its components is still on the CPU backend when Vulkan is expected. Even if you compiled with Vulkan support and saved the model, some ops or parameters might not have been properly transferred to the Vulkan backend before inference. You might want to double-check that you’re calling to (“vulkan”) on the model (or the inputs) before running inference.</p>

Cagatay_Bilgin: <p>Hey there, PyTorch Mobile is deprecated and we have been building ExecuTorch for the past few years. Check it out here: <a href="https://github.com/pytorch/executorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/executorch: On-device AI across mobile, embedded and edge for PyTorch</a> Vulkan support is better on ExecuTorch</p>
Any convient function to convert an index tensor to a tuple of index tensors in the first dimension?

chjz1024: <p>I have a compact D-dimensional index tensor <code>indices</code> of shape <code>(D, N)</code>. I would like to use them in <code>index_put</code> to update a container tensor <code>container</code>. In python this is simply achieved by <code>container.index_put(tuple(indices), values)</code>, however I have to use the following code to achieve that in c++:</p>
<pre data-code-wrap="cpp"><code class="lang-cpp">const std::vector&lt;std::optional&lt;at::Tensor&gt;&gt; tuple(const at::Tensor&amp; tensor) {
    std::vector&lt;std::optional&lt;at::Tensor&gt;&gt; vt;
    for(int i = 0; i &lt; tensor.size(0); i++) {
        vt.emplace_back(tensor[i]);
    }
    return vt;
}

... some stuff creating the container and values tensor.
    const c10::List&lt;std::optional&lt;at::Tensor&gt;&gt; x(tuple(indices));
    container.index_put(x, values);
</code></pre>
<p>I’m wondering if there exist some convenient methods to achieve this?</p>
Namespace "torch::jit" has no member "trace"

Sanjib: <p>I got the Namespace “torch::jit” has no member “trace” error while saving a custom trained model. Environments -</p>
<ol>
<li>libtorch-win-shared-with-deps-2.6.0+cu124</li>
<li>Visual Studio 2022 - Version- 17.10.2</li>
</ol>
<p>code snippet</p>
<pre><code class="lang-auto">try {   
    auto example_input = torch::rand({ 1, 3, img_height, img_width }).to(device);

    // Trace the model.
    auto traced_module = torch::jit::trace(*model, example_input);

    // Save the traced TorchScript module.
    traced_module.save(tensor_script_save_path);
    std::cout &lt;&lt; "Converted best model to " &lt;&lt; tensor_script_save_path &lt;&lt; " successfully." &lt;&lt; std::endl;
}
catch (const c10::Error&amp; error) {
    std::cerr &lt;&lt; "Error during TorchScript export: " &lt;&lt; error.what() &lt;&lt; std::endl;
    return -1;
}
</code></pre>
<p>Any suggestions?<br>
Thanks.</p>
Can a Lightning model be converted to TorchScript format?

evilroach: <p>I have trained my model by means of Lightning framework, and saved a checkpoint to a file.</p>
<p>Now I need to convert it into TorchScript format, in order to use it in C++ for producing.</p>
<p><strong>Is a Lightning checkpoint file in the same format as a regular Pytorch checkpoint?</strong><br>
Can I convert it into TorchScript format only according to the <a href="https://docs.pytorch.org/tutorials/advanced/cpp_export.html" rel="noopener nofollow ugc">tutorial</a>?</p>
<p>When I trying to do so, I get error as following:</p>
<blockquote>
<p>File “/home/leon/py-cpu/lib/python3.11/site-packages/torch/_sources.py”, line 33, in get_source_lines_and_file<br>
raise OSError(msg) from e<br>
OSError: Can’t get source for &lt;class ‘torch.cuda.FloatTensor’&gt;. TorchScript requires source access in order to carry out compilation, make sure original .py files are available.</p>
</blockquote>
<p>My code looks like:</p>
<pre><code class="lang-auto"># trainning on a server with CUDA(GPU)...

# loading on a PC only with CPU
model = Engine.load_from_checkpoint( 'path/to/checkpoint.file' )
# testing on PC
trainer.test( model, dataloaders=test_ld, verbose=True )
# No errors so far when reached here

# error occurs here
torch.jit.script( model ).save('torchscript_model.pt')
</code></pre>
<p>Environments:</p>
<ol>
<li>I define my model in another source file, not in the file calling to <code>torch.jit.script()</code>;</li>
<li>I trained it on my server with CUDA(GPU), but tried to convert it on my PC(CPU);</li>
</ol>
<p>Are these relavant?<br>
Thanks!</p>
How to customize leaf node in dynamo trace?

yushu.gao: <p>I am currently working on model quantization with graph export using <code>torch._dynamo.export</code> and have encountered issues with torch function call, let’s say it’s <code>torch.add</code>.</p>
<p>In the exported graph, we can only obtain the call_function node of <code>torch.add</code>. If there is more than one <code>torch.add</code> in the graph, users cannot set different qconfig for each <code>torch.add</code> unless they manipulate the exported graph directly, which is challenging. When I use FX symbolic trace, I could wrap the <code>torch.add</code> in an AddModule and make that AddModule a leaf node by override the is_leaf_module method of the CustomTracer. However, I haven’t found a similar feature in dynamo.</p>
<p>I have observed that the modules under the <code>torch.nn</code> namespace are represented as <code>call_modules</code> in the dynamo exported graph and I have attempted to define an <code>AddModule</code>, place it directly in the <code>torch.nn</code> namespace, and I can find the <code>AddModule</code>’s call_module node in the graph.</p>
<p>I believe there might be some configurations to make dynamo keep <code>call_module</code> instead of tracing into the forward of Module. I have tried changing configurations like <code>skipfiles_inline_module_allowlist</code> in <code>torch._dynamo.config</code> to instruct dynamo to treat <code>AddModule</code> as <code>call_module</code>, but these changes didn’t work.</p>
<p>So, my question is: Is there any way to customize the leaf node when using dynamo?</p>

twoflypig: <p>Same question here, is there someone can help?</p>

ice-tong: <p>I meet the same question~</p>
Acceleration for a loop

Smith_Jack: <p>Hello everyone. My program has the following layout</p>
<pre data-code-wrap="python"><code class="lang-python">for it in range(N, -1, -1):
    noise_of_A = model(A, it)
    A = A - noise_of_A
</code></pre>
<p>The <code>model</code> is very small, but N is large (300~500). The program runs slowly on GPU and the GPU usage rate is less than 15%. How can I accelerate it ?</p>

ptrblck: <p>Using CUDA Graphs will most likely help as it seems your CPU overhead is large compared to the GPU workload. You can either apply it manually or via <code>torch.compile</code> using the <code>'reduce-overhead'</code> option.</p>
How do I forcefully release the memory of a tensor?

NightMachinery: <p>I want to forcefully release a tensor’s memory. I don’t care if there are references alive to it.  Ideally, the tensor should throw an exception if it is accessed after being released.</p>
<p>I want this feature so that I release all tensor memory with a function like this:</p>
<pre><code class="lang-auto">def torch_gpu_remove_all():
    for obj in gc.get_objects():
        try:
            if torch.is_tensor(obj) or (
                hasattr(obj, "data") and torch.is_tensor(obj.data)
            ):
                if obj.is_cuda:
                    del obj
        except:
            pass
</code></pre>
<p>This will allow me to easily free up the memory without restarting the Jupyter kernel. I find this ability quite useful, especially when a GPU is being used intermittently by multiple users. This allows holding on to all state except those on a GPU, which is better than completely resetting the state.</p>

Vaen_Joe: <p>Are you sure the “del” operation will free tensor? “del” will trigger the “free” in CUDACachingAllocator only when the ref count was 0. Actually, “del” just make the ref count decreate 1.</p>
Flowers102 Train/Val/Test is 1020 / 1020 / 6149 samples instead of 6149 / 1020 / 1020?

dosssman: <p>Greetings.</p>
<p>I have been playing with the Oxford Flowers102 dataset with some diffusion script.<br>
The training was shorter than the code base I used as reference, and after checking the dataset length, it turned out that in torchvision (v3.6.0), the split is as follows:</p>
<p>Train: 1020 samples<br>
val: 1020 samples<br>
Test: 6149 samples</p>
<p>while in Hugginface’s dataset, it is<br>
Train: 7169 samples<br>
Test 1020 samples</p>
<p>I would have assume the train split takes most of the samples, so I was wondering if there was any specific reason for this data split ?</p>
<p>Thank you for your time.</p>
Using the trained model of torch in ipopt

j_tousi: <p>Hello.<br>
I’ve trained a model in PyTorch using <code>Dataset</code> and <code>DataLoader</code>. Now, I need to use this trained model within <strong>CyIpopt</strong>, which relies on <strong>NumPy arrays</strong> for gradient-based optimization. Since PyTorch uses tensors, I have to constantly convert data between NumPy and PyTorch formats. Additionally, I’ve been using a <code>DataLoader</code> for inference, assuming it’s necessary for accurate outputs.</p>
<p>However, repeatedly converting data types and creating a <code>DataLoader</code> in every iteration is significantly slowing things down, making real-time implementation questionable. Is there a more efficient way to integrate the PyTorch model with CyIpopt to improve performance?</p>

ptrblck: <aside class="quote no-group" data-username="j_tousi" data-post="1" data-topic="220202">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_tousi/48/77095_2.png" class="avatar"> j_tousi:</div>
<blockquote>
<p>However, repeatedly converting data types and creating a <code>DataLoader</code> in every iteration is significantly slowing things down, making real-time implementation questionable.</p>
</blockquote>
</aside>
<p>Are you cloning the array or are you using <code>torch.from_numpy()</code> to share the underlying data storage? The latter should be fast and I’m surprised if using a reference is already causing a bottleneck in your code.</p>
Torch.compile and Parameter replacement in a module

p.a_d: <p>Hi everyone,</p>
<p>I have a case where I need to manualy replace some parameters of a custom <code>nn.Module</code> at regular point during training.<br>
This looks essentially like this</p>
<pre data-code-wrap="python"><code class="lang-python">class Custom(torch.nn.Module):
    def __init__(self, ...):
        self.coef =  torch.nn.Parameter( ... initial values ...)

def updateCoef(module):
     module.coef = torch.nn.Parameter( someCalculation(module.coef) )
</code></pre>
<p>The function <code>updateCoef</code> is called when the parameter replacement is needed.</p>
<p>My question is what if I use <code>model.compile()</code> before the training ? Will the update of the parameter matters with respect to what is compiled ?<br>
I’ve done some limited testing and it seems there is some lag after the update, as if a recompilation was automatically triggered… but I’d like to be sure I’m not doing something plain wrong.</p>
<p>Thanks for any hints !</p>
Help: DDP hangs when using multi-threaded CUDA stream parallel loss computation

hantianyou: <p><strong>Body</strong>:</p>
<p>Hi everyone,</p>
<p>I’m training a multi-task model using PyTorch DistributedDataParallel (DDP), and I want to parallelize different loss computations (e.g., <code>pvb</code>, <code>lane3d</code>, <code>occ3d</code>, etc.) using multiple threads + custom CUDA streams for performance reasons.</p>
<p>I’ve implemented a <code>ParallelLossExecutor</code> class that looks roughly like this:</p>
<ul>
<li>Uses <code>ThreadPoolExecutor</code> with multiple threads</li>
<li>Each thread sets a specific <code>torch.cuda.Stream</code></li>
<li>Inside each thread, it runs the loss computation with:</li>
</ul>
<pre><code class="lang-auto">with torch.cuda.stream(stream):
    result = loss_fn(...)
stream.synchronize()
</code></pre>
<ul>
<li>After all losses are computed, I merge the loss dicts and do:</li>
</ul>
<pre><code class="lang-auto">total_loss = loss1 + loss2 + ...
total_loss.backward()
</code></pre>
<p>This works perfectly in <strong>single-GPU mode</strong>.</p>
<p><img src="https://discuss.pytorch.org/images/emoji/apple/red_exclamation_mark.png?v=14" title=":red_exclamation_mark:" class="emoji" alt=":red_exclamation_mark:" loading="lazy" width="20" height="20"> But in <strong>DDP multi-GPU mode</strong>, the program <strong>hangs during <code>backward()</code></strong>, most likely in <code>torch.distributed.all_reduce()</code>.</p>
<hr>
<h3><a name="p-470815-questions-1" class="anchor" href="#p-470815-questions-1"></a>Questions:</h3>
<ol>
<li>Is this kind of parallel loss computation (multi-thread + per-task CUDA streams) <strong>safe to use under DDP</strong>?</li>
<li>Does DDP assume that all autograd operations (including loss computation) happen in the <strong>default stream</strong> and <strong>main thread</strong>?</li>
<li>Are there any official recommendations on how to structure multi-task loss computations under DDP?</li>
</ol>
<hr>
<h3><a name="p-470815-other-info-2" class="anchor" href="#p-470815-other-info-2"></a>Other Info:</h3>
<ul>
<li>PyTorch version: [your version, e.g. 2.2.1]</li>
<li>NCCL backend</li>
<li>Mixed precision is off</li>
<li>All losses are computed from cloned model heads (to avoid parameter sharing issues)</li>
</ul>
<p>Thanks in advance! This is a pretty common pattern in multi-task learning, so I’d really appreciate any insight or recommendations.</p>
Pytorch running out of memory during inference

paegodu: <p>Hi all,</p>
<p>I am running a third-party model in inference mode as a calculator for ASE to optimize chemical structures. The issue is that after 66 our of ~200 structures PyTorch runs out of memory. I have been trying to track what happens by using torch.cuda.memory_allocated, torch.cuda.memory_reserved and torch.cuda.max_memory_reserved.</p>
<p>The weird thing is that while those show expected values of &lt;1GB of memory being used, after 66 iterations I get the following error:</p>
<blockquote>
<p>torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 496.00 MiB. GPU 0 has a total capacity of 79.10 GiB of which 48.75 MiB is free. Including non-PyTorch memory, this process has 79.04 GiB m     emory in use. Of the allocated memory 77.16 GiB is allocated by PyTorch, and 928.93 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALL     OC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (<a href="https://pytorch.org/docs/stable/notes/cuda.html#environment-variables" class="inline-onebox" rel="noopener nofollow ugc">CUDA semantics — PyTorch 2.7 documentation</a>)</p>
</blockquote>
<p>The main loop is inside of a “with torch.no_grad():”, only one instance of calculator (model) is used and torch.cuda.empty_cache() and torch.cuda.reset_peak_memory_stats() are called after every iteration (without calling the latter function torch.cuda.max_reserved_memory does scale and show an increasing value on par to what I would expect before the error message).</p>
<p>I am at a loss, where is that memory going? How do I release it?</p>
<p>Thanks!</p>

ptrblck: <aside class="quote no-group" data-username="paegodu" data-post="1" data-topic="220166">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/paegodu/48/77068_2.png" class="avatar"> paegodu:</div>
<blockquote>
<p>The weird thing is that while those show expected values of &lt;1GB of memory being used, after 66 iterations I get the following error:</p>
</blockquote>
</aside>
<p>The error shows 79GB are used so your manual memory checks might not be fine-grained enough and you could check if the model internally allocates this large amount of memory.</p>
Options for mobile inference

anAnnoyingNerd: <p>How can optimize a PyTorch model so I can run it on Android, without needing to convert the model to TorchScript?</p>

marksaroufim: <p>For now there aren’t all that many options, you can convert your model to torchscript, ONNX or tflite. But regardless of what you pick there will be some conversion process because phones don’t typically support python natively (iOS certainly doesn’t but perhaps with some workarounds it’s possible to do this in android)</p>

Cagatay_Bilgin: <p>There is a solid option now with ExecuTorch <a href="https://github.com/pytorch/executorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/executorch: On-device AI across mobile, embedded and edge for PyTorch</a> Join our discord channel at<br>
<a href="https://discord.gg/Wyn9PWqP" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Edge</a> where the team hangs out and can answer questions. We discuss a lot of the plans openly there.</p>

python: <p>Running PyTorch models on Android without conversion is tricky since phones don’t natively support Python. Most approaches, TorchScript, ONNX, and TFLite, involve some conversion.</p>
<p>A newer option is <strong>ExecuTorch</strong> (<a href="https://github.com/pytorch/executorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/executorch: On-device AI across mobile, embedded and edge for PyTorch</a>), which supports on-device inference with a subset of PyTorch. You can also join the PyTorch Edge Discord for more info.</p>
In inductor, can we unify the way to generate code between using extern_kernels and torch.ops?

Freerock_Neverdrop: <p>in pytorch inductor source code, there is the var named extern_kernels, which stores kernels to be called, also there is torch.ops, which is monkey patched to include ops there are called. Here the op and kernel are roughly the same thing. Strictly speaking here the kernel is actually an op.<br>
Some node are converted to ExternKernelSchedulerNode, and generate code like:</p>
<p>extern_kernels.addmm(primals_2, primals_3, reinterpret_tensor(primals_1, (2, 3), (1, 2), 0), alpha=1, beta=1, out=buf0)</p>
<p>While for ops, SchedulerNode is created, and generated code is like below:</p>
<pre><code>torch.ops.aten.bernoulli_.float(buf2, 0.8)
</code></pre>
<p>My question is, why not do it in the same way? e.g., treat addmm and an op and generate code like</p>
<p>torchops.aten.addmm()</p>
<p>We know that implementation of extern_kernels.addmm is actually aten.addmm.</p>
<p>What is the point to have the similar function duplicated?</p>

Sahoo: <p>How did you get these kind of ops like torch.ops.aten.bernoulli_ ? I either get the triton ops or extern_kernels ops?</p>
<p>I extern_kernels can link to external libraries as well.</p>
The nightly binary with CUDA 12.8

white711203: <p>I am currently using RTX5070TI. I know that if I want to use pytorch, I need to install the nightly binary version… The problem is that I need to use isaacgym/leggedgym which only supports python3.6~3.8. I currently cannot install the nightly binary version of torch in conda of python3.8… Can anyone help compile build the torch cu128 for the python3.8 version and supports rtx5070ti? Thank you…(my OS is ubuntu 22.04)</p>

ptrblck: <p>PyTorch binaries are built for Python 3.9-3.13 as older support was dropped. You could try to build PyTorch from source for Python 3.8 but I don’t know if that’s even possible with newer PyTorch releases.</p>

ahanjaya: <p>I tried to checkout tag: v2.4.1 for python3.8 and compile with TORCH_CUDA_ARCH_LIST=“12.0”.<br>
the output is:</p>
<pre><code class="lang-auto">nvcc fatal   : Unsupported gpu architecture 'compute_120'
</code></pre>

ptrblck: <p>Your CUDA toolkit raises this error as it’s too old as Blackwell architectures were added in 12.8. Besides that the PyTorch submodule is also too old as PRs enabling Blackwell landed right when 12.8 was releases (Jan/Feb 2025).</p>