akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Using nn.ParameterList

niata: <p>Hello,<br>
since I am using layers inside an array, they don’t get recognized by network.parameters() for the optimizer.<br>
To fix that I created a Parameterlist with the weights like this:</p>
<pre><code class="lang-auto">layer_size=[44,32,16,2]
self.layers=[torch.nn.Linear(layer_size[i] , layer_size[i+1]).to(device) for i in range(len(layer_size)-1)]
self.myparameters =  nn.ParameterList([nn.Parameter(p.weight) for p in self.layers])
</code></pre>
<p>Is this so correct that optim.Adam(self.network.parameters()) works right?</p>

Chirath_Pansilu: <p>I think it will be easier if you use <code>torch.nn.Sequential</code> container.</p>
<p>You can pass in your defined layer list to it and there will be no problem.</p>

niata: <p>Thanks, that works!<br>
Would the ParameterList create the same result (for example when I have a dropout, in an array i can easily disable it for test/evaluation, whereas i don’t see an easy solution for that in Sequential)?</p>

Chirath_Pansilu: <aside class="quote no-group" data-username="niata" data-post="3" data-topic="86742">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/7cd45c/48.png" class="avatar"> niata:</div>
<blockquote>
<p>whereas i don’t see an easy solution for that in Sequential</p>
</blockquote>
</aside>
<p>Actually you can disable dropout/batchnorm easily in this as well</p>
<pre><code class="lang-auto">model = nn.Sequential(*args)
model.eval()
</code></pre>
<p><code>model.eval()</code> puts the model into evaluation mode which disables dropout and use population statistics for batchnorm</p>

niata: <p>I made some tests between using a ParameterList and Sequencial and it seems to get the same results you either need to use weight and bias in the parameterlist in the same order or</p>
<p><code>self.layers=torch.nn.ModuleList([torch.nn.Linear(layer_size[i] , layer_size[i+1]).to(device) for i in range(len(layer_size)-1)])</code><br>
which seems easier.</p>

Peter78: <p>Dear all,</p>
<p>how can I remove an element from a ParameterList at runtime?</p>
<p>Regards,</p>
<p>P.</p>
[PR] Torchaudio incompatible with python flag -OO due to __doc__ being None

FremyCompany: <p>I opened the follwing pull request to fix the issue I think I’m facing.</p>
<p><a href="https://github.com/pytorch/audio/pull/3884" rel="noopener nofollow ugc">Avoid crash when <strong>doc</strong> is optimized away to None (-OO) by FremyCompany · Pull Request #3884 · pytorch/audio</a></p>
<p>Could someone take a look?</p>

ptrblck: <p>I would recommend avoiding creating cross-posts 5 mins after creating the PR and give the code owners a chance to take a look at it.</p>

FremyCompany: <p>I just followed the instructions from the <code>torchaudio</code> repository <img src="https://discuss.pytorch.org/images/emoji/apple/person_shrugging.png?v=12" title=":person_shrugging:" class="emoji" alt=":person_shrugging:" loading="lazy" width="20" height="20"></p>
<blockquote>
<p>PLEASE NOTE THAT THE TORCHAUDIO REPOSITORY IS NO LONGER ACTIVELY MONITORED. You will not likely get a response. For open discussions, visit <a href="https://discuss.pytorch.org/">https://discuss.pytorch.org/</a>.</p>
</blockquote>
Changing pytorch version in google colab

mohammad995: <p>I am trying to change the version of pytorch and torchvision to 1.7.1 and 0.8.2 in my colab notebook. I have tried several times but still it shows 2.1.0+cu121 for torch and 0.16.0+cu121 for torchvision, when I use torch.<strong>version</strong><br>
First I changed the python version to 3.7<br>
then I uninstall the current torch version using:<br>
!pip uninstall torch -y<br>
and then instal torch version 1.7.1 using:<br>
!pip install torch==1.7.1+cu101 torchvision==0.8.2+cu101 torchaudio==0.7.2 -f <a href="https://download.pytorch.org/whl/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/torch_stable.html</a><br>
what am I doing wrong?</p>

Jair_Ramos: <p>Hi, I have the same problem. Did you fix it? Thank you</p>
Using autograd to calculate taylor expansions

LeanderK: <p>Hi,</p>
<p>so I have a function <code>f: R^n -&gt; R</code> and I want to calculate the taylor expansion around <code>0</code> for all order up to total degree <code>d</code> (so 000, 001, …, 111, , (d-1)00, d00). I need to be more efficient, currently have some slow python loop using pytorch autograd. Is it possible to vectorize something here?</p>
<p>Naive code:</p>
<pre data-code-wrap="python"><code class="lang-python">
def multi_index_iterator(n, order):
    """
    Generate all multi-indices of dimension n with |alpha| &lt;= order.
    """
    for total_order in range(order + 1):
        for alpha in itertools.product(range(total_order + 1), repeat=n):
            if sum(alpha) == total_order:
                yield alpha


def taylor_coefficients(f, x0, order):
    """
    Compute Taylor coefficients for f at x0 up to a given order.
    
    Parameters:
    - f: a function f: R^n -&gt; R (expects a tensor input)
    - x0: a torch tensor of shape (n,) at which to expand (typically zero)
    - order: highest total derivative order
    Returns:
    - A dictionary mapping multi-indices (tuples) to their coefficient.
    """
    # Ensure x0 requires grad
    x0 = x0.clone().detach().requires_grad_(True)
    
    coeffs = {}
    # Loop over all multi-indices with |alpha| &lt;= order.
    for alpha in multi_index_iterator(len(x0), order):
        # Compute the derivative corresponding to alpha.
        # We'll use recursion: differentiate repeatedly according to the multi-index.
        # Define a helper function for recursive differentiation.
        def recursive_derivative(f, x, alpha, depth=0):
            if depth == len(alpha):
                return f(x)
            else:
                # Differentiate repeatedly with respect to x[depth]
                derivative = f(x)
                for _ in range(alpha[depth]):
                    grad = torch.autograd.grad(derivative, x, create_graph=True)[0]
                    derivative = grad[depth]
                # Move to next coordinate
                return recursive_derivative(lambda xx: derivative, x, alpha, depth + 1)
        
        derivative_val = recursive_derivative(f, x0, alpha).detach().item()
        # Divide by the factorial of the multi-index to get the coefficient.
        factorial = math.prod(math.factorial(a) for a in alpha)
        coeffs[alpha] = derivative_val / factorial
    return coeffs

</code></pre>
Python multiprocessing.Process hangs when large PyTorch tensors are initialised in both processes

jakelevi1996: <p>Why does the code shown below either finish normally or hang depending on which lines are commented/uncommented, as described in the table below?</p>
<p>Summary of table: if I initialise sufficiently large tensors in both processes without using <code>"spawn"</code>, the program hangs. I can fix it by making either tensor smaller, or by using <code>"spawn"</code>.</p>
<p>Note:</p>
<ol>
<li>All memory is purely CPU, I don’t even have CUDA installed on this computer</li>
<li>This issue does not occur if I replace <code>torch</code> with <code>numpy</code>, even if I make the array size 10x larger</li>
<li>Version information: <code>Ubuntu 22.04.1 LTS, Python 3.10.12, torch 2.1.2+cpu</code></li>
</ol>
<div class="md-table">
<table>
<thead>
<tr>
<th>Uncommented</th>
<th>Commented</th>
<th>Behaviour</th>
</tr>
</thead>
<tbody>
<tr>
<td>(1), (4)</td>
<td>(2), (3), (5)</td>
<td>Hang</td>
</tr>
<tr>
<td>(2), (4)</td>
<td>(1), (3), (5)</td>
<td>OK</td>
</tr>
<tr>
<td>(1), (5)</td>
<td>(2), (3), (4)</td>
<td>OK</td>
</tr>
<tr>
<td>(1), (3), (4)</td>
<td>(2), (5)</td>
<td>OK</td>
</tr>
</tbody>
</table>
</div><pre data-code-wrap="python"><code class="lang-python">import multiprocessing as mp
import torch

def train():
    print("start of train")
    x = torch.arange(100000)            # (1)
    x = torch.arange(10000)             # (2)
    print("end of train")

if __name__ == "__main__":
    mp.set_start_method('spawn')        # (3)
    x = torch.arange(100000)            # (4)
    x = torch.arange(10000)             # (5)
    p = mp.Process(target=train)
    p.start()
    p.join()
</code></pre>
Sub results different for tensor shape 1 and 32

guneshwors: <p>Why do torch.sub produce different results for the same data when shape is 1 and 32?</p>
<pre><code class="lang-auto">def fn(shape):
  alpha = -2.278090e-04
  x = torch.empty(shape, dtype=torch.float16).fill_(0.004044)
  y = torch.empty(shape, dtype=torch.float16).fill_(-17.39)

  print(torch.sub(x, y, alpha=alpha)[0])

fn(32)
fn(1)
</code></pre>
<p>prints</p>
<pre><code class="lang-auto">tensor(8.1837e-05, dtype=torch.float16)
tensor(8.0109e-05, dtype=torch.float16)
</code></pre>

ptrblck: <p>This is caused by the limited floating point precision and a different order of operations depending on the used algorithm as also seen e.g. in this simple example:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(100, 100)
s1 = x.sum()
s2 = x.sum(0).sum(0)
print(s1 - s2)
# tensor(1.5259e-05)
</code></pre>

guneshwors: <p>For elementwise, could you elaborate on</p>
<blockquote>
<p>different order of operations</p>
</blockquote>
<p>I was wondering if this is due to vectorized instructions used for shape &gt; 1.</p>

ptrblck: <p>Yes, maybe. You could check which kernel is called via a profiler for your use case exactly.</p>
How to Selectively Clear Gradients Accumulated from Previous Time Steps in RNN Backpropagation?

DawidL: <p>Hi,</p>
<p>I will try to explain my question through this toy problem:</p>
<p>Lets say I have a simple neural network consisting of a fully connected layer followed by an LSTM. The network will continuously receive inputs [X_1, X_2, X_3, …] and two iterations after each input, it is expected to output some value dependent on the input received 2 steps prior. In this case, the outputs will be [_, _, f(X_1), f(X_2), f(X_3), …], where _ indicates that we ignore the first two outputs, and f(X_n) is the output where the loss is based on the target that was paired with X_n.</p>
<p>However, the key difference in my approach is that I would like to backpropagate through the network after each output f(X_n), retaining only the gradients that contain information related to f(X_m) where m &gt; n. Hence, in my example, after backpropagating based on the output f(X_1), I would like to remove the gradient that was obtained in the first step. In other words, I would like to treat the hidden state that is fed into the LSTM at the same time as input X_2 as simply an input without any connections prior to that, and remove the gradients that were calculated for the fully connected layer after X_1 was passed through it, but before X_2 was passed through it.</p>
<p>I believe this can be partially solved by having multiple optimizers, however I think that would not work with standard optimizers like ADAM.</p>
DtoH transfer of a partial tensor or untyped_storage to tensor without memory allocation

aag: <p>Hi there!</p>
<p>I would like to convert a cupy function to torch: the purpose is to allocate pinned memory once, fill it with data (random size lower than allocated) and transfer it to the GPU.</p>
<p>i tried with untyped_storage but I failed creating a tensor pointing to the data_ptr of a pinned tensor and view is not possible because of shape incompatibility.<br>
constraints:<br>
no more memory allocations than what is done with cupy<br>
no use of deprecated functions</p>
<p>here is the cupy code</p>
<pre data-code-wrap="python"><code class="lang-python">max_nbytes = ...
h_pinned_mem = cp.cuda.alloc_pinned_memory(max_nbytes) # memory allocation 1 at the initialization (cpu)

# loop, threaded etc, whatever...
variable_buffer_nbytes = ... # from 1 to max_nbytes
...
random_buffer = np.random.bytes(variable_buffer_nbytes) # memory allocation 2 (cpu)
h_pinned_array: np.ndarray = np.frombuffer(h_pinned_mem, dtype=np.uint8, count=variable_buffer_nbytes)
np.copyto(h_pinned_array, random_buffer)

d_cp_tensor = cp.empty((variable_buffer_nbytes,), cp.uint8) # memory allocation 3 (gpu), in my sw it's done at the initialization and never allocated after that
d_cp_tensor.set(h_pinned_array)
d_torch_tensor: torch.Tensor = from_dlpack(d_cp_tensor.toDlpack())
</code></pre>
<p>and here is some code in torch</p>
<pre data-code-wrap="python"><code class="lang-python">h_pinned_tensor = torch.empty(
    max_nbytes, dtype=torch.uint8, device='cpu', requires_grad=False, pin_memory=True
)
# fill  h_pinned_tensor with random_buffer, size=variable_buffer_nbytes

untyped_storage: UntypedStorage = bytes_tensor.untyped_storage()
partial_untyped_storage = untyped_storage[0:variable_buffer_nbytes]

# Create a tensor pointing to partial_untyped_storage
# complete this
h_partial_pinned_tensor = .... # &lt;- must not allocate memory, just point to untyped_storage and size must be variable_buffer_nbytes
#

d_torch_tensor: torch.Tensor = h_partial_pinned_tensor.to("cuda")
</code></pre>
<p>=&gt; Is it possible or should I have to stick with CuPy?</p>
<p>ps: It’s not needed to ask Claude, chatgpt, they’re wrong.<br>
i don’t need any advice about memory overflow check, cuda streams, cuda synchronisation etc. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
SSL error: "certificate verify failed" for downloading the MNIST dataset

make1234: <p>Hello,</p>
<p>I want to build a simple neural network but get an SSL error and google searches did not yield a solution. The error is the following and the line is the one below my comment in the source code:</p>
<pre><code class="lang-auto">Error downloading train-images-idx3-ubyte.gz:
Tried http://yann.lecun.com/exdb/mnist/, got:
HTTP Error 404: Not Found
Tried https://ossci-datasets.s3.amazonaws.com/mnist/, got:
&lt;urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)&gt;
</code></pre>
<pre><code class="lang-auto">import torch
from torch.utils.data import DataLoader
from torchvision import datasets
from torchvision.transforms import ToTensor
from utils import *

import ssl

ssl._create_default_https_context = ssl._create_stdlib_context

"""
Here we fetch the dataset. 
"""
# the error is in this line below.
train_set = datasets.MNIST(root="data", download=True, transform=ToTensor())
test_set = datasets.MNIST(root="data",
                          download=True,
                          transform=ToTensor(),
                          train=False)
"""
Here we have specified a batch size of 64. 
"""
train_loader = DataLoader(train_set, batch_size=32, shuffle=False)
test_loader = DataLoader(test_set, batch_size=32, shuffle=False)
"""
These lines of code simply start iterating through the dataloader, visualize
the first 4 images of the first batch, and then terminate the iteration.
"""
for batch, labels in train_loader:
    print(
        """Tensor dimension of a single batch
(batch_size x channels x height x width):""", list(batch.shape))
    visualize_first_4(batch, labels)
    break
</code></pre>
<p>I would be very grateful if someone knew a solution <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

make1234: <p>I found the error - a ./ was missing before data.</p>
How to Perform Multiple Inferences on Multiple GPUs in PyTorch?

dmp58: <p>Hi,</p>
<p>I need to perform inference using the same model on multiple GPUs inside a Docker container. I want to run inference on multiple input data samples simultaneously across different GPUs within the Docker environment, so each GPU is processing a different input batch in parallel using the same model.</p>
<p>However, it seems that the GPUs are not being fully utilized. While one GPU is processing, the other seems to be idle. I would like to ensure that both GPUs are being used simultaneously for inference and that the workload is distributed effectively.</p>
<p>To clarify, my goal is to:</p>
<ul>
<li><strong>Replicate the same model</strong> across multiple GPUs.</li>
<li><strong>Distribute input batches</strong> across GPUs in parallel.</li>
<li>Ensure both GPUs are actively processing during inference without one resting.</li>
</ul>
<p>Any help or guidance on how to best set this up using PyTorch inside Docker would be greatly appreciated!</p>

ptrblck: <p>You could try to implement a custom multi-processing solution or use inference libs, such as <a href="https://pytorch.org/serve/">torchserve</a> or <a href="https://developer.nvidia.com/triton-inference-server">Triton Inference Server</a>.</p>

dmp58: <p>Hi,</p>
<p>I am already using torch.multiprocessing for parallel inference, but I still notice that when one GPU is performing inference, the other remains idle. My goal is to have both GPUs running inference simultaneously on different batches. Is there something I might be missing in my setup?</p>
Pytorch mtcnn.detect()

Aleksa_Novevski: <p>I am using facenet_pytorch and I’ve been using its mtcnn to detect faces. It worked properly, now i wanted to get vectors from larger dataset of images (around 300) and a lot of them give me None as face is not recognized, where the image is of a very good quality and clearly a face.<br>
I use it like:<br>
from facenet_pytorch import MTCNN, InceptionResnetV1</p>
<p>mtcnn = MTCNN()<br>
img = Image.open(img_path).convert(‘RGB’) -Image from PIL library<br>
boxes, _ = mtcnn.detect(img)</p>
<p>And not for all but some of them I get boxes to be None, i go to the file system. Very nice image, preprocessed, face seen clearly, good lightning… Any clues?</p>

Hasnan_Ramadhan: <p>same as me, when the mtcnn in cuda and image move to cuda to with transform .tensor first. The image can not be read return none tensor in mtcnn(img). But when I see my image.shape is return a value</p>

yassineAlouini: <p>Could you share one of the failing images so that we can help you debug?</p>
Converting model that has dynamic control flow to load in C++

firebolt: <p>Hello,</p>
<p>I’m working on converting a PyTorch model to a format that can be loaded in C++. My model contains dynamic control flow (for loops and if blocks), so I’ve been using TorchDynamo and TorchInductor for optimization. However, I’m encountering difficulties in finding a way to load these optimized models in C++.</p>
<p>I’ve tried the following approaches:</p>
<ol>
<li>Using TorchDynamo and TorchInductor to optimize the model</li>
<li>Attempting to use TorchScript on the optimized model</li>
<li>Exploring AOT Inductor to convert the model to .pt format</li>
</ol>
<p>Unfortunately, none of these methods have been successful in producing a .pt file that can be loaded in C++.</p>
<p>My questions are:</p>
<ol>
<li>Is it currently possible to load TorchDynamo/TorchInductor-optimized models with dynamic control flow in C++?</li>
<li>If so, what is the recommended approach or workflow to achieve this?</li>
</ol>
Embedding model GPU memory usage

Jared4: <p>Hi,</p>
<p>I am loading this embedding model on to the GPU memory.</p>
<p>The model has total Parameters: <strong>335,141,888</strong><br>
The precision that I am using to load the model is <strong>bfloat16</strong>.<br>
When I check the memory from nvidia-smi, it is using approximately 1.25GB .</p>
<p>I was expecting to model to use <strong>0.625 GB</strong> (model parameters x precision, 2 bytes for bf16)</p>
<p>The code is:<br>
from transformers import AutoModel<br>
model_name = “BAAI/bge-large-en-v1.5”<br>
model = AutoModel.from_pretrained(model_name, torch_dtype=torch.bfloat16).to(“cuda”)</p>
<p>Why the model is using using this additional memory?<br>
Appreciate any help.</p>

ptrblck: <p>Parameters are stored in <code>float32</code>, which matches the memory usage, unless you explicitly cast them to <code>bfloat16</code>. I don’t know what <code>AutoModel</code> does.</p>
ImportError: cannot import name 'fail_with_message' from 'torchaudio._internal.module_utils'

kjhyioa0: <p>ImportError: cannot import name ‘fail_with_message’ from ‘torchaudio._internal.module_utils’ (/home/lab/miniconda3/envs/train2/lib/python3.8/site-packages/torchaudio/_internal/module_utils.py)</p>
<p>My CUDA version is 11.6. So I install this version.</p>
<pre><code class="lang-auto"># CUDA 11.6
conda install pytorch==1.13.0 torchvision==0.14.0 torchaudio==0.13.0 pytorch-cuda=11.6 -c pytorch -c nvidia
</code></pre>
<p>What’s the problem?</p>

ptrblck: <p>This method was merged in <a href="https://github.com/pytorch/audio/pull/2968">this PR</a> on Jan 12, 2023, so should show up starting with <code>torchaudio&gt;=2.0</code> based on the <a href="https://github.com/pytorch/audio/releases?page=2">release dates</a>.</p>
Error trying to import function from torchtext

EvanZ: <p>I have setup an Amazon Linux EC2 gpu instance with Python 3.11.6 and am trying to import torch and torch text:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; from torchtext.vocab import build_vocab_from_iterator
</code></pre>
<p>This raises the following error:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/home/ec2-user/***/.env/lib64/python3.11/site-packages/torchtext/__init__.py", line 6, in &lt;module&gt;
    from torchtext import _extension  # noqa: F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ec2-user/***/.env/lib64/python3.11/site-packages/torchtext/_extension.py", line 64, in &lt;module&gt;
    _init_extension()
  File "/home/ec2-user/***/.env/lib64/python3.11/site-packages/torchtext/_extension.py", line 58, in _init_extension
    _load_lib("libtorchtext")
  File "/home/ec2-user/***/.env/lib64/python3.11/site-packages/torchtext/_extension.py", line 50, in _load_lib
    torch.ops.load_library(path)
  File "/home/ec2-user/***/.env/lib64/python3.11/site-packages/torch/_ops.py", line 1350, in load_library
    ctypes.CDLL(path)
  File "/usr/lib64/python3.11/ctypes/__init__.py", line 376, in __init__
    self._handle = _dlopen(self._name, mode)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /home/ec2-user/***/.env/lib/python3.11/site-packages/torchtext/lib/libtorchtext.so: undefined symbol: _ZN5torch3jit17parseSchemaOrNameERKSs
</code></pre>
<p>I’ve seen other people have run into this issue when they have multiple versions of torch installed, but this instance doesn’t have anything else installed, so I’m at a loss. Any ideas?</p>

ptrblck: <p>Could you share the PyTorch and <code>torchtext</code> versions you are using?</p>

EvanZ: <p>Trying to use torch 2.6.0 (2.5.1 gave the error as well) and torchtext 0.18.0.</p>

EvanZ: <p>Just to be complete, it looks like I’m getting the same error importing any function or class (like Vocab) from torchtext, or even torchtext itself.</p>

ptrblck: <p>Thank you for the clarification!<br>
I’m afraid this regression might be expected since <code>torchtext</code> does not seem to be developed anymore. From <a href="https://github.com/pytorch/text">pytorch/text</a>:</p>
<blockquote>
<p><strong>WARNING</strong>: TorchText development is stopped and the 0.18 release (April 2024) will be the last stable release of the library.</p>
</blockquote>
<p>The <code>Installation</code> section mentions PyTorch nightly builds should be compatible with <code>torchtext:main</code>, but I haven’t tested it.</p>
convert the weight deit_transreid_veri.pth from pth to onnx #92

mootaz_haddad: <p>dummy_input = torch.randn(1, 3, 256, 256).to(device)<br>
dummy_cam = torch.zeros(1, dtype=torch.long).to(device)<br>
dummy_view = torch.zeros(1, dtype=torch.long).to(device)<br>
torch.onnx.export(<br>
model,<br>
(dummy_input, dummy_cam, dummy_view),<br>
“deit_transreid_veri.onnx”,<br>
input_names=[‘input’, ‘cam_label’, ‘view_label’],<br>
output_names=[“output”],<br>
dynamic_axes={<br>
‘input’: {0: ‘batch_size’},<br>
‘cam_label’: {0: ‘batch_size’},<br>
‘view_label’: {0: ‘batch_size’},<br>
‘output’: {0: ‘batch_size’}<br>
},<br>
opset_version=13,<br>
verbose=True,<br>
)</p>
<p>print(“Model has been successfully exported to ONNX format!”)</p>
<p>when i run the code onnx export stop without error and no onnx file is generated<br>
can you help me ?</p>
Executorch -QNN backend

vinay_perumalla: <p>I have locally compiled llama 3.2-3b model followed steps in this link <a href="https://github.com/pytorch/executorch/blob/main/examples/demo-apps/android/LlamaDemo/docs/delegates/qualcomm_README.md" class="inline-onebox" rel="noopener nofollow ugc">executorch/examples/demo-apps/android/LlamaDemo/docs/delegates/qualcomm_README.md at main · pytorch/executorch · GitHub</a></p>
<p>Converted model to .pte with below command</p>
<h1><a name="p-464817-h-4-bits-weight-only-quantize-1" class="anchor" href="#p-464817-h-4-bits-weight-only-quantize-1"></a>4 bits weight only quantize</h1>
<p>python -m examples.models.llama.export_llama --checkpoint “${MODEL_DIR}/consolidated.00.pth” -p “${MODEL_DIR}/params.json” -kv --disable_dynamic_shape --qnn --pt2e_quantize qnn_16a4w -d fp32 --metadata ‘{“get_bos_id”:128000, “get_eos_ids”:[128009, 128001]}’ --output_name="test.pte”</p>
<p>The model give stutter response<br>
Prompt:HI<br>
Response:”,!!!</p>
<p>Which not expected please help me how can i get the correct response</p>

jackzhxng: <p><a class="mention" href="/u/shoumikhin">@shoumikhin</a> <a class="mention" href="/u/cccclai">@cccclai</a></p>

cccclai: <aside class="quote no-group" data-username="vinay_perumalla" data-post="1" data-topic="216275">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vinay_perumalla/48/75485_2.png" class="avatar"> vinay_perumalla:</div>
<blockquote>
<p>Which not expected please help me how can i get the correct response</p>
</blockquote>
</aside>
<p>Yeah it needs to be calibrated properly to have a reasonable result. SpinQuant is an option, if you follow <a href="https://pytorch.org/executorch/stable/llm/build-run-llama3-qualcomm-ai-engine-direct-backend.html" class="inline-onebox" rel="noopener nofollow ugc">Building and Running Llama 3 8B Instruct with Qualcomm AI Engine Direct Backend — ExecuTorch 0.5 documentation</a></p>
<p>however the rotation matrics isn’t available out of box either and you’d need to run <a href="https://github.com/facebookresearch/SpinQuant" class="inline-onebox" rel="noopener nofollow ugc">GitHub - facebookresearch/SpinQuant: Code repo for the paper "SpinQuant LLM quantization with learned rotations"</a> to get them</p>
<p>We understand it’s quite a pain now and will try to make it easier</p>

vinay_perumalla: <p>Is there away we can enable 3.2-3b , and you mention about calibration of model in above comment could you share the exact steps to export it in to .pte</p>
Inference with float16

ytesfai: <p>Hi,</p>
<p>I have a Linear/GRU NN implementation that will eventually be in an embedded system. As such I need the inference to be done with float16. I tried performing the test with model.half() but I got the following error:</p>
<p>*** RuntimeError: “compute_indices_weights_linear” not implemented for ‘Half’</p>
<p>I read some discussions in this regard pointing out that float16() is not supported in a CPU as it wouldn’t much speedup. However in my case I want to test with float16 to make sure there is no performance degradation when I port my network to embedded environment with all float16. Is there any way for me to do this other than moving to a CUDA for my inference test?</p>
<p>Thanks<br>
Yohannes</p>

ptrblck: <p>If you are indeed seeing this issue on the CPU, then your concern might be valid and you would need to either use a GPU for <code>float16</code> support or you could try to use <code>bfloat16</code> instead on the CPU or GPU.</p>

ytesfai: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thanks for the recommendations. I tried bfloat16, it also had it’s own issues. I went with testing on a GPU and that works.</p>
<p>Not sure what the protocol is for requesting features but for anyone considering embedded NN inference, smaller data size like float16 I think is very important so it would be nice if it was fully supported.</p>
Doubt about how to properly use 3D affine transformation

sebamenabar: <p>Hi, let’s say I have the grid <em>grid</em>, a 3D representation, of size <em>(size, size, size)</em> <img src="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b08fbe16c776d78410b568fbec61d61330e69f14.png" alt="Captura de pantalla 2020-03-20 a la(s) 13.36.10" data-base62-sha1="pbW06Lu6G9f6YeM22JwFsITCDWI" width="147" height="128"> and I’d like to apply some rotation, scaling and translation (R, S, T) to it (all 4x4 in homogenous coordinates, T = [Identity(4,3) | t], Identity(4,3) is and identity matrix of 4 rows and 3 columns and t a vector of size 4 with 1 in its last position).</p>
<p>The equivalent transformation is defined as<br>
<code>theta = torch.bmm(torch.bmm(T, S), R)</code></p>
<p>To generate the sampling positions I make<br>
<code>sample_grid = affine_grid(theta, (batch_size, num_channels, new_size, new_size, new_size))</code><br>
But according to the <a href="https://github.com/pytorch/pytorch/blob/74b65c32be68b15dc7c9e8bb62459efbfbde33d8/aten/src/ATen/native/AffineGridGenerator.cpp" rel="noopener nofollow ugc">implementation</a> this does this:</p>
<pre><code class="lang-auto">Tensor base_grid = make_base_grid_5D(theta, N, C, D, H, W, align_corners);
auto grid = base_grid.view({N, D * H * W, 4}).bmm(theta.transpose(1, 2));
</code></pre>
<p>And from my understanding of Multiview Geometry (wihch is very scarce so there’s I high chance I’m wrong), this calculates where a point in the new grid would be mapped to in the old grid, by applying the transformation theta. But since what I want is to apply a rotation, scaling and translation to the original grid, I’d have to use the inverse of that <code>theta</code> to achieve it.<br>
Math explanation</p>
<pre><code class="lang-auto"># The final position (x', y', z') as applying a rotation R, scaling S and translation T to a point (x,y,z) can be computed as:
# \theta * (x, y, z) = T * S * R * (x, y ,z) = (x', y', z')
# Multiplying by the inverses
# (x, y, z) = R^-1 * S^-1 * T^-1 * (x', y', z')
</code></pre>
<p>This way I correctly apply the transformation to my original grid, and not the other way arround (the transform to the new grid)</p>
<p>Final question: Do I use <code>theta</code> or <code>theta^-1</code> to create the sampling grid?</p>
<p>Please let me know if I failed to explain clearly my doubt and if I’m wrong the reasons.<br>
Thanks!</p>

orion: <p>It is surprising that no one in the community has come up with an answer for this. The simplest question might be this: what does theta imply for 3D affine transformation? Is it the multiplication of the rotation and translation matrices or what? The source code also does not provide a step-by-step implementation, which puts us in the dark. I would love to hear some answers regarding this issue.</p>
Understanding Runtime Drops in PyTorch Backward Pass and Efficient Implementations

Ece: <p>Hello everyone,</p>
<p>I am currently working on reinforcement learning experiments using a neural network and a differentiable physics simulator. My goal is to analyze the effects of different gradient modification methods on the training process. To implement these modifications, I need to perform both modified and original forward and backward passes using PyTorch.</p>
<p>To improve efficiency, I attempted to replace two separate forward-backward passes with a single forward pass, followed by computing both original and modified gradients using loops over the computational graph. Theoretically, this should be more efficient, as it reduces redundant computations. However, in practice, I observed that these “efficient” implementations take more time than the original version, likely due to PyTorch’s optimized backward pass.</p>
<p>Additionally, as seen in the attached graph, the runtime does not always increase monotonically. Instead, at certain timesteps, there are sharp drops in runtime for both the original and modified implementations. Interestingly, when the “efficient” implementation experiences a drop, it actually becomes faster than the original version.</p>
<p>I have a few questions:</p>
<ol>
<li>What could be causing these sudden runtime drops in the backward pass?</li>
<li>Are there any high-level (PyTorch API) or low-level (CUDA, autograd internals) optimizations that could help achieve these drops consistently?</li>
<li>Would restructuring my computation graph or using different autograd functions improve efficiency in this case?</li>
</ol>
<p>I would really appreciate insights from anyone with experience in PyTorch autograd internals or computational graph optimizations. Thank you in advance for your time and help!</p>
<p>(Attached: Graph illustrating the runtime behavior of different implementations averaged over 20 runs. Efficient 1 and Efficient 2 are 2 different versions for efficient computation graph.)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b593d38fe03759d7e0c4ab12ee58bbaa793443c.png" data-download-href="https://discuss.pytorch.org/uploads/default/1b593d38fe03759d7e0c4ab12ee58bbaa793443c" title="pytorch_runtime_results"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b593d38fe03759d7e0c4ab12ee58bbaa793443c.png" alt="pytorch_runtime_results" data-base62-sha1="3TW5jqYrqHjpoHByYqEMDGZpQAI" width="450" height="278"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch_runtime_results</span><span class="informations">600×371 37.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Time to analyze, problem hardware ? or libraries?

yadira: <p>I have this hardware 11th Gen intel i5 2.40Ghz, 32 GB RAM, Disk ssd nvme and GeForce rtx 4090, and libraries Pytorch, difussers, accelerate and python 3.10, I have already made images with SDXL and now I’m trying load FLUX either Dev or Schnell the issue is that it takes a long time to load the steps for image generation, the pipeline loads very fast but it takes a long time even though I only add 5 steps. I have tested this configuration on a 4070 Ti and it also takes time but the average is between 7 to 9 minutes for three images, but in the case of the 4090, it takes up to 20 or 30 minutes. this is a very basic code but maybe you can get an idea of what to change.</p>
<pre><code class="lang-auto">print(torch.__version__)  # Torch version: 2.6.0+cu118
    print(torch.cuda.is_available())  #  True
    
    global pipe_flux
    torch.cuda.empty_cache()
    model_id_flux = "C:/Users/GFAdmin/Documents/model/flux.1-dev/"
    torch.set_default_tensor_type(torch.cuda.FloatTensor)
    torch.cuda.set_device(0)
    #
    pipe_flux = FluxPipeline.from_pretrained(
        model_id_flux, 
        torch_dtype=torch.bfloat16
    )
    pipe_flux.to("cuda")
    pipe_flux.reset_device_map() 
    pipe_flux.enable_model_cpu_offload()

    print("Pipeline FLUX por defecto cargado.")
    
    torch.cuda.empty_cache()
    torch.cuda.reset_peak_memory_stats()
    with autocast("cuda"), torch.inference_mode():
        image = pipe_flux(
                "A capybara holding a sign that reads Hello World",
                num_inference_steps=5,
                guidance_scale=3.5,
            ).images[0]
        image.save("./pruebas/capybara.png")
</code></pre>
<p>and another case I have other code used checkpoint file but not working too</p>
<pre><code class="lang-auto">import torch
    from diffusers import FluxPipeline, AutoencoderKL
    from transformers import T5EncoderModel
    torch.cuda.set_device(0)

    model_file = "C:/Users/GFAdmin/Documents/model/flux_checkpoint/flux_dev.safetensors"
    text_encoder = CLIPTextModel.from_pretrained("C:/Users/GFAdmin/Documents/model/clip-vit-base-patch32")
    text_encoder_2 = T5EncoderModel.from_pretrained("t5-base")
    vae = AutoencoderKL.from_pretrained("stabilityai/sd-vae-ft-ema")
    

    pipe_flux = FluxPipeline.from_single_file(
        model_file,
        torch_dtype=torch.float16,
        use_safetensors=True,
        text_encoder=text_encoder,
        text_encoder_2=text_encoder_2,
        vae=vae
    )

    pipe_flux.to("cuda")
    pipe_flux.reset_device_map()
    pipe_flux.to(torch.float16)
    prompt = "A capybara holding a sign that reads Hello World"
    result = pipe_flux(prompt, num_inference_steps=5, guidance_scale=3.5)
    result.images[0].save("./pruebas/capybara.png")

    print("Imagen guardada en './pruebas/capybara.png'")

</code></pre>
<p>I will be grateful for any help</p>

ptrblck: <p>I would recommend profiling your code via e.g. Nsight Systems to narrow down the bottleneck of your application.</p>
Vision mamba implementation

Omaralmaqtari: <p>I am a new user of the Mamba model, I read a few papers saying it has a robust performance on image segmentation tasks. If someone has implemented it before, is there any guidance on the correct setup for the mamba block or input image patches and dimensions that could lead to the best results?</p>
<p>My implementation so far has not shown any advantage of adding the Mamba block into my code, here is a small snippet of my implementation:</p>
<pre><code class="lang-auto">x = torch.rand(1, 16, 256, 256)
norm = RMSNorm(16**2)
mamba = Mamba(16**2)
_, c, h, _ = x.shape
x = rearrange(x, 'b c (p1 ph) (p2 pw) -&gt; b (c p1 p2) (ph pw)', ph=16, pw=16)
x = mamba(norm(x)) + x
x = rearrange(x, 'b (c p) d -&gt; b c p d', c=c)
x = rearrange(x, 'b c (p1 p2) (ph pw) -&gt; b c (p1 ph) (p2 pw)', p1=h//16, ph=16)
</code></pre>
`__origin__` `AttributeError` when using `torch.library.custom_op`

Avi_Chapman: <p>When I try to use <code>torch.library.custom_op</code> to decorate an existing function in the <code>spconv</code> library, I get an error <code>AttributeError: __origin__</code>.</p>
<p>The method I am decorating is an exact copy of the <code>ops.get_indices_pairs</code> method. I’ve added return typehinting and a metakernal.</p>
<p>My code looks like this:</p>
<pre><code class="lang-auto">@torch.library.custom_op("mylib::get_indice_pairs", mutates_args=())
def get_indice_pairs(...) -&gt; Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
   ...

@get_indice_pairs.register_fake
def _(...) -&gt; Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
  return torch.empty(), torch.empty(), torch.empty()
</code></pre>
<p>Can anyone shed any light on this? The fact that it is looking for <code>__origin__</code> would seem to indicate to me that it is about a generics issue, but I have no idea how.</p>
Built-in 0-1 loss function?

Geremia: <p>PyTorch have a built-in 0-1 loss function?</p>
<p><a href="https://www.deeplearningbook.org/contents/ml.html#pf7" rel="noopener nofollow ugc">Goodfellow et al. p. 102</a>:</p>
<blockquote>
<p>The 0-1 loss on a particular example is 0 if it is correctly classified and 1 if it is not.</p>
</blockquote>
<p>A custom 0-1 loss function could easily be implemented:</p>
<pre data-code-wrap="python"><code class="lang-python">class ZeroOneLoss(torch.nn.Module):
    def __init__(self):
        super(ZeroOneLoss, self).__init__()

    def forward(self, input, target):
        if input.equal(target):
            return torch.tensor(0., requires_grad=True)
        else:
            return torch.tensor(1., requires_grad=True)
</code></pre>

KFrank: <p>Hi Geremia!</p>
<aside class="quote no-group" data-username="Geremia" data-post="1" data-topic="216747">
<div class="title">
<div class="quote-controls"></div>
<img alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/geremia/48/70165_2.png" class="avatar"> Geremia:</div>
<blockquote>
<p>PyTorch have a built-in 0-1 loss function?</p>
</blockquote>
</aside>
<p>No.</p>
<aside class="quote no-group">
<blockquote>
<p>A custom 0-1 loss function could easily be implemented:</p>
</blockquote>
</aside>
<p>A 0-1 loss function would not be (usefully) differentiable.  That is, after backpropagation,<br>
the gradients computed for a model’s parameters would all be zero (hence giving the<br>
optimizer no information about how to modify those parameters to reduce the loss).</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">class ZeroOneLoss(torch.nn.Module):
    def __init__(self):
        super(ZeroOneLoss, self).__init__()

    def forward(self, input, target):
        if input.equal(target):
            return torch.tensor(0., requires_grad=True)
        else:
            return torch.tensor(1., requires_grad=True)
</code></pre>
</blockquote>
</aside>
<p>This <code>if-else</code> construct is not differentiable.  Setting <code>requires_grad = True</code> for the<br>
return value doesn’t fix this.</p>
<p>(You could write a soft loss that is close to one when <code>input</code> and <code>target</code> are nearly<br>
equal, but is close to zero when <code>input</code> and <code>target</code> differ significantly.)</p>
<p>Best.</p>
<p>K. Frank</p>

Geremia: <p><a class="mention" href="/u/kfrank">@KFrank</a> Thanks. I was going to write an answer to my own question and say something similar: that this function is not differentiable, so is useless for backprop.</p>
How to convert pth to onnx?

aatefi: <p>Hi,</p>
<p>I used Pointpillars for 3D object detection on my own dataset (just *.bin files and their *.txt labels) using Pytorch (Cuda). The format of the trained model is *.pth. I would like to convert the *.pth file to *.onnx format. I think I should define a model (Pointpillars) and a dummy tensor/input in this regard. I am not sure how to define them. Is there any example/library for Pointpillars in this regard?<br>
I appreciate any help in advance.</p>
<p>Thank you,<br>
Abbas</p>

Thomas_Waheed: <p>Did you find any solutions?</p>

Sokmontrey_Sythat: <p>You can load the <code>.pth</code> file into a model and export it as <code>.onnx</code>.</p>
<p>Load the model:</p>
<pre data-code-wrap="py"><code class="lang-py"># ... model definition
model.load_state_dict(t.load("./model.pth"))
</code></pre>
<p>Save the model to onnx</p>
<pre data-code-wrap="py"><code class="lang-py">t.onnx.export(model, t.randn(28 * 28), "./model.onnx") 
</code></pre>
<p>Please note that <code>t.randn(28*28)</code> is a sample input (you might need to include batch size) to properly generate the onnx graph.</p>
<p>Hope this help,<br>
Sokmontrey</p>
Does torch.cuda.Event influence performance?

elefant: <p>I am currently using <code>torch.cuda.Event(enable_timing=True)</code> for measuring the forward, backward and total iteration time of DNN training. This data is used to predict the iteration time of a DNN training setup. I noticed that when I enable profiling, these timings are a little longer than expected, and consequently the prediction is higher than my baseline experiments. I derived the expected value from a tensorboard profiling report.</p>
<p>I am using the following context manager to measure the time of a code chunk:</p>
<pre><code class="lang-auto">@contextlib.contextmanager
def take_gpu_time() -&gt; ContextManager[None]:
    start_event = torch.cuda.Event(enable_timing=True)
    end_event = torch.cuda.Event(enable_timing=True)
    record = DurationRecord(start_event, end_event)
    start_event.record()
    yield record
    end_event.record()
</code></pre>
<p>The getter of <code>DurationRecord.duration</code> looks as follows:</p>
<pre><code class="lang-auto">@property
def duration(self):
    torch.cuda.synchronize()
    start_event.synchronize()
    end_event.synchronize()
    return start_event.elapsed_time(end_event)
</code></pre>
<p>With this asynchronous design, I hope to gain a lesser impact on performance by deferring the device synchronization to a point in time when I actually need the result (i.e., after the iteration completed).</p>
<p>Could this code have a notable impact on the performance of DNN training, especially when it’s called often (i.e., during each forward and backward call)?</p>
Question About the Design of Serialization Function Signature in BackendMeta.

Seungchul_Han: <p>The serialization function of BackendMeta requires metadata to be serialized into a std::unordered_map&lt;std::string, bool&gt;. While this works well for simple boolean states, it introduces complexity when serializing other data types, such as integers. For instance, if I need to store an integer value, I have to convert it into a string or encode it in a way that fits into this structure, which adds extra overhead during serialization and deserialization.</p>
<ol>
<li>
<p>Why was std::unordered_map&lt;std::string, bool&gt; chosen instead of a more flexible structure?</p>
</li>
<li>
<p>Is the primary intent of this function to handle only simple presence/absence states, or is it expected to support more complex metadata types with additional processing?</p>
</li>
</ol>
<p>Understanding the reasoning behind this design will help me better align with PyTorch’s intended usage patterns and contribute effectively.</p>
<p>Thank you for your time and insights.</p>
Inference_mode() doesn't remove grad

blmppes: <p>Hi guys as I know inference_mode() will remove grad, but in my case here it doesn’t → so I have to use y_preds.detach().</p>
<p>Anyone has any ideas about this?<br>
Thanks for your help!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/d/1d5da9ab3a51d653e4e1cc79efce5660959de40d.png" data-download-href="https://discuss.pytorch.org/uploads/default/1d5da9ab3a51d653e4e1cc79efce5660959de40d" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d5da9ab3a51d653e4e1cc79efce5660959de40d_2_690x439.png" alt="image" data-base62-sha1="4bMw7XAoipFwwD2s41oIiwuMSyp" width="690" height="439" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/d/1d5da9ab3a51d653e4e1cc79efce5660959de40d_2_690x439.png, https://discuss.pytorch.org/uploads/default/original/3X/1/d/1d5da9ab3a51d653e4e1cc79efce5660959de40d.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/d/1d5da9ab3a51d653e4e1cc79efce5660959de40d.png 2x" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">909×579 29.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue, please?</p>