akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Cuda GPU never found

rXp: <p>Environment :</p>
<ul>
<li>3050 Ti</li>
<li>Cuda 12.5 (latest drivers)</li>
<li>Windows 11</li>
<li>Miniconda 3</li>
</ul>
<p>I’ve been trying to make PyTorch Cuda work for two days but I have been unable to.</p>
<blockquote>
<p>torch.cuda.is_available()</p>
</blockquote>
<p>Will always return false.</p>
<p>Nvidia-smi returns<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/19327070558ddbbdec436b1e8dca3aa624312b79.png" data-download-href="https://discuss.pytorch.org/uploads/default/19327070558ddbbdec436b1e8dca3aa624312b79" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/9/19327070558ddbbdec436b1e8dca3aa624312b79.png" alt="image" data-base62-sha1="3AU08L9o5CV4ETwRQZOEFCBGQWt" width="690" height="301" data-dominant-color="1B1B1B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1296×566 18.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have simply ran this command :</p>
<blockquote>
<p>conda install pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia</p>
</blockquote>
<p>I have tried different versions (12.1, 12.4) and also tried to reinstall my NVidia drivers or the miniconda environment.<br>
I am trying to run this on my laptop which has 2 GPU, one from the NVIDIA card with Cuda and one from the AMD CPU/GPU for low energy consumption.</p>
<p>When I try to get the environment information it is missing Cuda for Pytorch :</p>
<blockquote>
<p>Collecting environment information…<br>
PyTorch version: 2.3.0<br>
Is debug build: False<br>
CUDA used to build PyTorch: Could not collect<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Microsoft Windows 11 Pro<br>
GCC version: Could not collect<br>
Clang version: Could not collect<br>
CMake version: Could not collect<br>
Libc version: N/A</p>
<p>Python version: 3.12.3 | packaged by Anaconda, Inc. | (main, May  6 2024, 19:42:21) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)<br>
Python platform: Windows-11-10.0.22631-SP0<br>
Is CUDA available: False<br>
CUDA runtime version: 12.5.40<br>
CUDA_MODULE_LOADING set to: N/A<br>
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3050 Ti Laptop GPU<br>
Nvidia driver version: 555.85<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A<br>
Is XNNPACK available: True</p>
</blockquote>
<p>I can run software that uses Cuda without issue (cuda-z or a few graphics softwares) so I know the drivers doesn’t have an issue.</p>
<p>The only time I managed somehow to have the environment setup properly, any Cuda calls would hang forever.</p>
<p>Is there a problem of version ? What can I do to make this work ?</p>
Leverage graph mode computation for pure c++ nn.module

Mat32: <p>Hi everyone, I might have missed that answer… I have a nn.module defined in c++ and I want to leverage graph mode computation to both improve computation time and hide its implementation.<br>
AFAIK there’s no way to use jit::trace in c++ to export it to TorchScript.</p>
<p>There is thus little options left. Is embedding TorchScript code into c++ source code a viable solution? I didn’t find any example to get started.<br>
Or is there other options ?</p>
<p>Thanks</p>
Requirements for using torch with RTX 4070

DickC: <p>I have a new Dell XPS 9640 with a NVIDIA RTX 4070. I have installed torch v 2.2.2 and Python 3.12.2, and am using the latest version of VSCode. a simple Python script indicates GPU is not available. What am I missing here?</p>

ptrblck: <p>Assuming you have installed the correct PyTorch binary with CUDA support, you might want to check your NVIDIA drivers and reinstall them if needed.</p>

DickC: <p>when I show torch in terminal it reports:<br>
Name: torch<br>
Version: 2.2.2<br>
Summary: Tensors and Dynamic neural networks in Python with strong GPU acceleration<br>
Home-page: <a href="https://pytorch.org/" rel="noopener nofollow ugc">https://pytorch.org/</a><br>
Author: PyTorch Team<br>
Author-email: <a href="mailto:packages@pytorch.org">packages@pytorch.org</a><br>
License: BSD-3<br>
Location: c:\Users\rdcap\pythonScripts.venv\Lib\site-packages<br>
Requires: filelock, fsspec, jinja2, networkx, sympy, typing-extensions<br>
Required-by: torchaudio, torchvision</p>
<p>nvidia reports:</p>

ptrblck: <p>What does <code>torch.__version__</code> and <code>torch.version.cuda</code> return?</p>

DickC: <p>torch.<strong>version</strong> says 2.2.2+cpu (oops! shouldn’t it say +GPU?) and if so, how do I get the right version</p>

ptrblck: <p>Yes, it should show a valid CUDA runtime version. Refer to <a href="https://pytorch.org/get-started/locally/">the install instructions</a>. The easiest way would be to simply run <code>pip install torch</code>, which will install the latest stable PyTorch release with CUDA 12.1.</p>

DickC: <p>i installed torch with pip install torch torchvision torchaudio --extra-index-url <a href="https://download.pytorch.org/whl/cu124/torch_stable.html" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu124/torch_stable.html</a></p>

ptrblck: <p>This install command is invalid and not listed in the support matrix.<br>
The CUDA 12.4 builds are still being worked on, so stick to the provided install commands.</p>

DickC: <p>Fixed! I backed up to cuda 12.1 and the gpu is available. thanks for your help</p>

John_Fitzgerald: <p>Hey guys,</p>
<p>Just a note of thanks. This info was very helpful.</p>
Pytorch pretrained models using contrastive learning on ImageNet

Giang_Nguyen: <p>Hello folks,</p>
<p>I am running a project and it requires contrastive features. I am aware of many deep learning models trained with contrastive learning paradigm out there but could not find any pretrained models which I can leverage the pretrained features for my problem.</p>
<p>I am working on ImageNet.</p>
<p>Any suggestions are appreciated.<br>
Thanks so much!</p>

ANUMULA_CHAITANYA_SA: <p>why dont you train your model using a supervised contrastive loss function on imagenet ,if not possible at least you can train on tinyimagenet</p>
Puzzled about the learning rate(lr) for Adam

Kobeyond: <p>Hey guys, recent I was quite puzzled about the learning rate for Adam optimizer in Pytorch:<br>
In many demos I have read, it seems that the proper value of lr for Adam is usually 0.01 or 0.001. However, when I set lr=0.001 in my code, the training loss and accuracy will increasing and decreasing violently all the time,  and never converge at all.<br>
So, only when I set lr around <strong>0.00001</strong>, the training process becomes to be  normal, and finally my model works well ! Though the problem has been solved, I think the current value of lr is still too much small =_=.<br>
Is this phenomenon normal? If not, what caused it? I am waiting sincerely for you guys to solve my puzzle!!!</p>

ANUMULA_CHAITANYA_SA: <p>Hi, I think it totally depends on the task at hand and the data you are using, if your loss cure is steep everywhere, then smaller updates in the gradient will be needed</p>
Using opset version higher than 18 in onnx

skanh: <p>Hi, I want to use layers that are only implemented in ONNX opset version 20 and higher.</p>
<p>However, <code>onnx.export()</code> cannot be used because it supports up to opset version 17 only. I’m trying to use <code>onnx.dynamo_export()</code>, but it returns a warning saying, ‘torch.onnx.dynamo_export only implements opset version 18 for now. If you need to use a different opset version, please register them with register_custom_op.’</p>
<p>I don’t understand how to use <code>register_custom_op</code>. My model includes GELU; how can I ensure the GELU layer is exported correctly?</p>
<p>Thank you.</p>

dennisleon: <p>Hello,</p>
<p>Unfortunately, using register_custom_op to export a model with GELU and a higher opset version than 18 is not currently supported in PyTorch.</p>
<p>Here are some alternative approaches you can consider:</p>
<ol>
<li>Upgrade PyTorch:</li>
</ol>
<p>Check if you’re using the latest stable version of <a href="https://www.health-insurancemarket.com" rel="noopener nofollow ugc">Official Site</a> PyTorch. Newer versions might have better support for higher opset versions and GELU export.<br>
2. Use a custom function:</p>
<p>You can define a custom function that replicates the GELU behavior using existing ONNX operators supported by your desired opset version. This might involve using erf or other mathematical approximations.<br>
3. Wait for future updates:</p>
<p>The PyTorch developers are actively working on extending dynamo_export to support higher opset versions and more operators like GELU. Keep an eye on future releases for potential improvements.<br>
4. Consider alternative frameworks:</p>
<p>If exporting to ONNX with higher opset versions is crucial, you might need to explore other frameworks like TensorFlow or ONNX Runtime that offer better support for newer opset versions and potentially have native GELU implementations.<br>
Additional notes:</p>
<p>While register_custom_op exists, it’s intended for advanced use cases and requires deep understanding of ONNX operators and their implementation details.<br>
Using a custom function can be a more straightforward approach if you’re comfortable with defining and implementing the GELU behavior using existing ONNX operators.</p>
<p>I hope the information may helps you.</p>
Error: No activations detected in Lightning module with Opacus

fsommers: <p>I’ve followed the MNIST Lightning example in the examples directory of the Opacus repo.</p>
<p>But I’m implementing a module that uses DistilBERT for an NLP classification task, instead of my own custom network.</p>
<p>Inside my module’s <strong>init</strong>(), I define:</p>
<p><code>self.model = DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased', num_labels=num_labels)</code></p>
<p>My <code>forward()</code> simple does the following:</p>
<pre><code class="lang-auto">def forward(self, input_ids, attention_mask, labels=None):
    return self.model(input_ids=input_ids, attention_mask=attention_mask, labels=labels)
</code></pre>
<p>As in the example, I defined <code>make_private</code> in the configure_optimizers() function of my module.</p>
<p>The only difference is that to obtain the data loader inside configure_optimizers(), I follow the updated way of doing this with Lightning:</p>
<pre><code class="lang-auto">if self.enable_dp:
  self.trainer.fit_loop.setup_data()
  data_loader = self.trainer.train_dataloader
</code></pre>
<p>When I run the Lightning trainer’s fit() method:</p>
<p><code>trainer.fit(model, train_loader, val_loader)</code></p>
<p>I get the following error:</p>
<pre><code class="lang-auto">/usr/local/lib/python3.10/dist-packages/opacus/grad_sample/grad_sample_module.py in rearrange_grad_samples(self, module, backprops, loss_reduction, batch_first)
    381         """
    382         if not hasattr(module, "activations"):
--&gt; 383             raise ValueError(
    384                 f"No activations detected for {type(module)},"
    385                 " run forward after add_hooks(model)"

ValueError: No activations detected for &lt;class 'torch.nn.modules.linear.Linear'&gt;, run forward after add_hooks(model)
</code></pre>
<p>My Lightning module works fine without Opacus, so I’m not sure what I’m doing wrong.</p>
<p>Thanks in advance.</p>
Fine-tune LLMs using torchtune

kostya.loparev: <p>Hello!<br>
Due to the release of Llama3, I wanted to fine-tune it using the torchtune on rtx4090, but ran into 2 problems. Perhaps you can help solve them:</p>
<ol>
<li>when starting training, torchtune gives an error:<br>
RuntimeError: bf16 precision was requested but not available on this hardware. Please use fp32 precision instead.<br>
In the same time: “torch.cuda.is_bf16_supported()” return True</li>
<li>if i specify dtype fp32, another error occurs:<br>
torch._C._nn.cross_entropy_loss<br>
RuntimeError: expected scalar type Long but found Int</li>
</ol>

pparshakov: <p>Got same error. RTX 3090, torch.cuda.is_bf16_supported() gives True.</p>

marksaroufim: <p>Are you using Windows by any chance? There’s a known issue there otherewise this is unexpected since 4090 does indeed support bf16 <a href="https://github.com/pytorch/torchtune/issues/891" class="inline-onebox">Runtime Error: BF16 unsupported on supported hardware · Issue #891 · pytorch/torchtune · GitHub</a></p>

kostya.loparev: <p>I corrected the code a little and launched the training on Windows. If this is still relevant to someone, I will describe my actions</p>

yutang0105: <p>I got same error on 4090.</p>

pparshakov: <p>It is very relevant for me <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

pparshakov: <p><a class="mention" href="/u/kostya.loparev">@kostya.loparev</a> can you share your solution?</p>

kartikayk: <p>Can you try using the nightly build as opposed to the stable package? The instructions are here:<br>
<a href="https://pytorch.org/torchtune/main/install.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/torchtune/main/install.html</a></p>
<p>We’ll have a package push in the next couple of weeks to make sure the fix is in the stable package.</p>
What's the use of `torch.bool` tensors?

Geremia: <p>What’s the use of <a href="https://pytorch.org/docs/stable/tensors.html#id5" rel="noopener nofollow ugc"><code>torch.bool</code></a> tensors, if most layers use floats?</p>
<p>I see the smallest bitwidth tensor besides <a href="https://pytorch.org/docs/stable/tensors.html#id5" rel="noopener nofollow ugc"><code>torch.bool</code></a> is <a href="https://pytorch.org/docs/stable/tensors.html#id6" rel="noopener nofollow ugc"><code>torch.quint4x2</code></a> (unsigned 4-bit integer), but only one sort of layer (<a href="https://pytorch.org/docs/stable/generated/torch.nn.EmbeddingBag.html#torch.nn.EmbeddingBag" rel="noopener nofollow ugc"><code>nn.EmbeddingBag</code></a> <a href="https://pytorch.org/docs/stable/nn.html#sparse-layers" rel="noopener nofollow ugc">sparse layer</a>) supports it.</p>
<p>Pytorch’s docs on “<a href="https://pytorch.org/docs/stable/quantization.html" rel="noopener nofollow ugc">Quantization</a>” define it as:</p>
<blockquote>
<p>techniques for performing computations and storing tensors at lower bitwidths than floating point precision.</p>
</blockquote>
<p>But as low as 1-bit?</p>
<p><sup>See <a href="https://ai.stackexchange.com/q/45864/48038" rel="noopener nofollow ugc">my related AI.StackExchange question</a>.</sup></p>

ptrblck: <p>You can use it e.g. as a masking tensor:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(10)
print(x)
# tensor([ 0.0662,  0.6553,  0.7496,  0.2320, -0.0108, -0.2568,  0.7380,  0.3812,
#          0.2378,  1.4405])

mask = x &gt; 0.
print(mask)
# tensor([ True,  True,  True,  True, False, False,  True,  True,  True,  True])
print(mask.dtype)
# torch.bool

y = x[mask]
print(y)
# tensor([0.0662, 0.6553, 0.7496, 0.2320, 0.7380, 0.3812, 0.2378, 1.4405])
</code></pre>
Model returns a tuple instead of tensor

babak_abad: <p>I wrote a program in python using pytorch. My problem is:<br>
my <strong>model</strong> returns <strong>tuple instead of tensor</strong>. How can I make/force my model to return tensor?</p>
<p>this my code:</p>
<pre><code class="lang-auto">import torch
from torch import nn
import numpy as np
from torch.optim import Adam

x = np.linspace(0, 1, 50).reshape((-1, 1)).astype('float32')
y = np.power(x, 2).reshape((-1, 1)).astype('float32')

x = torch.tensor(x)
y = torch.tensor(y)

model = nn.Sequential(
    nn.Linear(1, 1),
    nn.ReLU()
)

loss = nn.MSELoss()
opt = Adam(model.parameters(), lr=0.001)
model.train()
n_batch = 4
n_epoch = 100
for i in range(n_epoch):
    for b in range(0, len(x), n_batch):
        inp = x[b:b+n_batch]
        out = y[b:b+n_batch]
        opt.zero_grad()
        pred = model(inp),
        ls = loss(pred, out)
        ls.backward()
        opt.step()
</code></pre>
<p>This the last lines of error that I got:</p>
<blockquote>
<p>File “D:\Projects\Sine_Wave_LSTM_Pytorch\venv\lib\site-packages\torch\nn\functional.py”, line 3355, in mse_loss<br>
if not (target.size() == input.size()):<br>
AttributeError: ‘tuple’ object has no attribute ‘size’</p>
</blockquote>

ptrblck: <aside class="quote no-group" data-username="babak_abad" data-post="1" data-topic="203866">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/babak_abad/48/69464_2.png" class="avatar"> babak_abad:</div>
<blockquote>
<p>my <strong>model</strong> returns <strong>tuple instead of tensor</strong>.</p>
</blockquote>
</aside>
<p>Remove the comma after <code>model(inp)</code>:</p>
<aside class="quote no-group" data-username="babak_abad" data-post="1" data-topic="203866">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/babak_abad/48/69464_2.png" class="avatar"> babak_abad:</div>
<blockquote>
<p><code>        pred = model(inp),</code></p>
</blockquote>
</aside>
How can I define a new data type of tensor or set grad to bool tensor?

fangwei123456: <p>I want to use bool tensor with float grad. But in default, the data type of tensor’s grad is same with the tensor’s data.</p>
<pre><code class="lang-auto">x = torch.rand([8]) &gt; 0.5
x.requires_grad_(True)
</code></pre>
<p><code>RuntimeError: only Tensors of floating point dtype can require gradients</code></p>

ptrblck: <p>The gradient <code>dtype</code> is equal to the tensor’s data type as it’s used to update the data.<br>
Could you explain a bit what a floating point gradient on a bool value would mean and how a parameter update would work?</p>

fangwei123456: <p>Thanks for your reply! In fact, I want to use a tensor with values in {0, 1}, which acts like being quantized to binary. For example:</p>
<pre><code class="lang-python">device = 'cuda:0'
x = torch.as_tensor([0., 0., 1., 1.], device=device)
x.requires_grad_(True)
y = torch.as_tensor([0., 1., 0., 1.], device=device)
y.requires_grad_(True)
z = x * y  # z = x AND y
print(z)
z.sum().backward()
print(x.grad)
print(y.grad)
</code></pre>
<p>For the moment, I use float tensor to implement such a kind of tensor. But when values in tensors are binary, sotring them in float is not necessary. Sotring binary values in bool can reduce memory cost. So, I want to store values in bool, and store gradient in float.</p>

Geremia: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="140604">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Could you explain a bit what a floating point gradient on a bool value would mean and how a parameter update would work?</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/fangwei123456">@fangwei123456</a> Yes, how would derivatives/gradients be performed on boolean tensors?<br>
This seems related to my question here “<a href="https://discuss.pytorch.org/t//203871">What’s the use of <code>torch.bool</code> tensors?</a>” (or <a href="https://ai.stackexchange.com/q/45864/48038" rel="noopener nofollow ugc">my related AI.StackExchange question</a>).</p>

ptrblck: <p>Boolean tensors do not support gradients (same as integer types) as the gradient would be zero everywhere and undefined or Inf at the rounding points.</p>
Can libtorch return torch::Tensor to pytorch with pybind11？

dancedpipi: <p>I need do something on libtorch, then return Tensor to pytorch, implement by pybind11</p>
<p>code snip：</p>
<pre><code class="lang-auto">#include &lt;torch/torch.h&gt;
#include &lt;pybind11/pybind11.h&gt;
#include &lt;iostream&gt;

torch::Tensor create()
{
    return torch::rand({2, 3});
}


PYBIND11_MODULE(example, m)
{
    m.doc() = "libtorch tensor test";
    m.def("create", &amp;create, "create tensor");
}
</code></pre>
<p>when I execute “example.create()” in Python, error occur:</p>
<blockquote>
<p>TypeError: Unable to convert function return value to a Python type! The signature was<br>
() → at::Tensor</p>
</blockquote>
<p>How can I do that correctly? Hope for your help!</p>

dancedpipi: <p>I guess it is because “torch::Tensor” is not bound in pybind11, but how to bind it? And make the bound “torch::Tensor” and “pytorch tensor” the same type?</p>

tom: <p>Yes, PyTorch can do it, but you need to import the right header, I think</p>
<pre><code class="lang-auto">#include &lt;torch/extension.h&gt;
</code></pre>
<p>will do the trick.</p>
<p>The background is that the Python bits are the thing you would not want to include when you are using libtorch purely in C++, so you want more than just <code>torch/torch.h</code> if you use Python.</p>
<p>Best regards</p>
<p>Thomas</p>

dancedpipi: <p>Thanks for your reply, are you saying that as long as the torch/extension.h file is included? is there any example for reference?</p>

dancedpipi: <p>when I only include torch/extension.h, error occur while import in python</p>
<blockquote>
<p>ImportError: /home/work/test_libtorch/build/example.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _Z16THPVariable_WrapN2at6TensorE</p>
</blockquote>

dancedpipi: <p><a class="mention" href="/u/tom">@tom</a>  Oh I see the <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">tutorial</a>, it’s helpful!</p>

take-cheeze: <p>From my experience you need to <code>import torch</code> before loading the extension module unless you link libtorch_python.so to the extension</p>

adizhol: <p>I also get ImportError mypacakge.cpython-38-x86_64-linux-gnu.so: undefined symbol: _Z16THPVariable_WrapN2at10TensorBaseE<br>
I added <span class="hashtag">#include</span> &lt;torch/extension.h&gt; in my cpp file, but still it doens’t work</p>

tttzof351: <p>For history:</p>
<p>Fix here: <a href="https://github.com/pytorch/pytorch/issues/108041#issuecomment-1753895665" class="inline-onebox" rel="noopener nofollow ugc">Undefined Symobl: pybind11::detail::type_caster&lt;at::Tensor, void&gt;::load(pybind11::handle, bool) · Issue #108041 · pytorch/pytorch · GitHub</a></p>
Can't reproduce same output with einsum

hibama: <p>Hello. I was conducting a few experiments to see if changing the order of matrix multiplications in Linear Attention would yield the same results.</p>
<pre><code class="lang-auto">import torch

"""input and weights"""
in_original = torch.randn(1, 197, 192).to(torch.device("cpu"))
q_linaer = torch.randn(192, 192).to(torch.device("cpu"))
k_linaer = torch.randn(192, 192).to(torch.device("cpu"))
v_linear = torch.randn(192, 192).to(torch.device("cpu"))


q = torch.matmul(in_original, q_linaer)
k = torch.matmul(in_original, k_linaer)
v = torch.matmul(in_original, v_linear)

"""split"""
B, C, L = in_original.shape
q = q.reshape(B, 3, C, -1)
k = k.reshape(B, 3, C, -1)
v = v.reshape(B, 3, C, -1)

"""in_qw_kv"""
kv = torch.einsum("b h j c, b h j d -&gt; b h c d", k, v)
q_linaer = torch.reshape(q_linaer,(B, 3, L, -1))
qwkv = torch.einsum("b h i c, b h c d -&gt; b h i d", q_linaer, kv)
re_out = torch.einsum("b c i, b h i d -&gt; b h c d", in_original, qwkv)
re_out = torch.reshape(re_out, (B, C, L))

"""qk-v"""
qk = torch.einsum("b h i c, b h j c -&gt; b h i j", q, k)
mhsa = torch.einsum("b h i j, b h j d -&gt; b h i d", qk, v)
mhsa = torch.reshape(mhsa, (B,C,L))
original_output = mhsa


"""q-kv"""
kv = torch.einsum("b h j c, b h j d -&gt; b h c d", k, v)
qkv = torch.einsum("b h i c, b h c d -&gt; b h i d", q, kv)
qkv = torch.reshape(qkv, (B, C, L))

print(qkv-mhsa)
print(re_out-mhsa)


</code></pre>
<pre><code class="lang-auto">tensor([[[ 0.0469, -0.0625,  0.0352,  ...,  0.0469, -0.0312,  0.1094],
         [ 0.1406,  0.0391,  0.0781,  ...,  0.0781, -0.0312, -0.0781],
         [ 0.0156,  0.0000, -0.1250,  ...,  0.0625,  0.0312,  0.0469],
         ...,
         [-0.0586,  0.0312,  0.0625,  ...,  0.0156, -0.1250, -0.0312],
         [-0.1094,  0.1250,  0.0625,  ...,  0.0625, -0.1172,  0.0625],
         [ 0.0625, -0.0312, -0.0234,  ..., -0.0469,  0.1875, -0.0703]]])
tensor([[[ 261114.0625, -860036.0000, -656980.0000,  ...,  731625.5625,
           279413.4688,  -20652.5312],
         [ 494674.3125,  -74906.0938, -359076.0000,  ...,  181925.0469,
           415182.0625,  458257.1250],
         [-115711.3594,  366081.9375,  -10439.5625,  ...,  -73224.4688,
           -25387.6250, -303460.0000],
         ...,
         [  -5849.0781,  280915.9375,  473818.0938,  ...,  -29121.1484,
           670491.7500, -327646.3438],
         [-152076.0781, 1076525.0000, -985963.1250,  ..., -353680.5000,
          -321628.1250,  271734.7812],
         [ -72602.5312,  372593.5000, -612792.2500,  ..., -144943.8438,
          -260349.6875,  465726.4062]]])

Process finished with exit code 0

</code></pre>
<p>Upon checking the above code, I found that <code>qkv</code> and <code>mhsa</code> have the same values, but <code>re_out</code> shows different values from <code>mhsa</code>.<br>
What could be the issue?</p>

hibama: <p>torch.reshape was the problem changing it to einops.rearrange solved it</p>
Input size and sequence length for lstm

abhishek_bh: <p>I have music data which contains raw vocals of some indian classical and devotional songs and I have segmented each file into 20 seconds files. I have used librosa.load function with a sample rate of 3000 to load the audio file. The output returned by the function is of type ndarrray and I have converted it to pytorch tensor of type float and reshaped it to(1,60001). Each audio file is labeled. I want to do the classification task using lstm. If I consider batch_size as 32 then input will be of shape(32,1,60001). I want to know what is the input size and sequence length in this case. Considering audio as a single channel input will the input size be 1 and sequence length be 60001.The model ran faster each epoch when the input size was 60001.Model ran very very slow when the input size was 1.Which is the correct way of considering the input size and sequence length of the lstm model. Can lstm be trained using raw audio files for classification task??</p>

bokki: <p>ya ya it can be trained. <img src="https://discuss.pytorch.org/images/emoji/apple/crazy_face.png?v=12" title=":crazy_face:" class="emoji" alt=":crazy_face:" loading="lazy" width="20" height="20"></p>
Conv2D forward and backward pass very slow

Jersack: <p>Hello,</p>
<p>I am using python 3.10.13 and pytorch 2.2.2+cu121</p>
<p>I updated a small part of my network and basically changed a MLP with a multi head attention, my input is of shape (Batch, Channel, Sequence_length, Embedding_dim). To do so I had to add 2 Convolutional layer, 1 at the entrance of the Attention layer to collapse the channel dimension into 1 (B, C, S, E) → (B, 1, S, E) → (B, S, E) and a last one to recreate the channel dimension and continue with the rest of the model (B, S, E) → (B, 1, S, E) → (B, C, S, E)</p>
<p>My issue is I noticed a huge slowdown of the network (no matter if gpu/cpu/multi-gpu). On a GPU, an epoch used to take a minute and now takes 40 minutes.<br>
I found out the forward pass is still taking the same amount of the time but the backward pass became 10x slower than the forward instead of 1-2x slower as before. After manually evulating the Conv2D and attention layer I noticed the Conv2D were incredibly much slower despite their much less number of parameters (less than a hundred vs millions for the attention layer)</p>
<p>To get the numbers below I just created the layers and used random inputs and outputs to do the forward pass and to compute a dummy loss for the backward pass.<br>
On CPU, the Conv2D at the entrance (C → 1) takes ~0.004 ms forward pass and takes ~0.040 backward pass so the backward pass is 10x times slower. The CNN at the exit (1 → C) takes ~0.05 ms forward pass and ~0.10 ms backward pass so backward pass is 2 times slower (which is normal) but both forward and backward are very slow and should have been in the order of ~0.001ms ?</p>
<p>The attention layer takes as an input shape (S, E) and both forward/backward pass is ~0.03ms</p>
<p>Both Conv2D don’t even have more than 100 parameters so why does it takes so much time to forward/backward those layers than it takes to forward/backward an attention layer or a mlp with millions of parameters ?</p>
<p>Best</p>

Jersack: <p>So I tested my code a bit more and apparently the input size is the one and only culprit here</p>
<p>With an input size of (B, 50,4) the backward pass is 1.5x slower than forward pass (so it’s normal).<br>
With an input size of (B,50, 600) the backward pass is 10x slower than forward pass.<br>
My model is just a Unet</p>
<p>Is this 10x slowdown normal ?! I tried torch.compile() and even though it is faster the ratio seems the same.<br>
I’ve trained many different architectures before and it’s the first time I see such a difference between the forward and backward pass, even with the same number of paramaters</p>
Multiple workers in iterable dataset

Tlotlo_Oepeng: <pre data-code-wrap="python"><code class="lang-python">
class TrackMLIterableDataset(IterableDataset):
    def __init__(self, data_path):
        self.data_path = data_path

    def __iter__(self):
        worker_info = torch.utils.data.get_worker_info()
        if worker_info is None:
            return self._generate_data()
        else:
            worker_id = worker_info.id
            num_workers = worker_info.num_workers
            return self._generate_data(worker_id, num_workers)

    def _generate_data(self, worker_id=0, num_workers=1):

        for event_id, hits, cells, particles, truth in load_dataset(self.data_path):
            if (event_id % num_workers) == worker_id:
                particles = particles[particles['nhits'] &gt;= 5]
                merged_df = pd.merge(truth, particles, on='particle_id')
                merged_df = pd.merge(merged_df, hits, on='hit_id')

                merged_df['pT'] = np.sqrt(merged_df['px']**2 + merged_df['py']**2)

                grouped = merged_df.groupby('particle_id')

                for particle_id, group in grouped:
                    inputs = group[['tx', 'ty', 'tz']].values
                    target = group[['pT']].values[0]

                    input_tensor = torch.tensor(inputs, dtype=torch.float32)
                    target_tensor = torch.tensor(target, dtype=torch.float32)

                    mask = torch.ones(input_tensor.shape[0], dtype=torch.bool)
                    yield (input_tensor, mask, target_tensor)
</code></pre>
<p>Im trying to use  workers in my data loading scheme, this is the best i could come up with but its not idea not to mention slower, what are possible schemes that i could try?</p>
<p>To explain the code i loop through different scv csv files to make one training example,</p>
Nearing Completion, Seeking Guidance on LSTMCell and Optimizer Conversion (TensorFlow to PyTorch)

Hrridoyv2: <p>I’m in the final stages of meticulously converting a TensorFlow model to PyTorch. While I’ve made significant progress, I’m encountering a specific hurdle in translating the <code>LSTMCell</code> method and optimizer implementation.<br>
<strong>The code links:</strong> <a href="https://colab.research.google.com/drive/1J-lbaAA_MTMedLkwXMDPrWD70OsXDiHA#scrollTo=x13oz2R3i6xO" rel="noopener nofollow ugc">Tensorflow Code Link</a>  …    <a href="https://colab.research.google.com/drive/1avEqBg2qFyJCV05UBCnEoh_7dgYQqQHC#scrollTo=iOlBHnD5i-KV" rel="noopener nofollow ugc">Pytorch Code Link</a></p>
What is meant by already allocated memory in cuda gpu?

Md_Minhazur_Rahman: <p>My dataset is 20.0gb in total. I am using A100 8gpus each has 40gb memory. Why my inference is failing? And what is meant by that  “21.09 GiB already allocated” message??? I see this already allocated thing for all gpus.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/9/090ab5c9d3ad7cb1cedd56e227fa496e30e85ec4.png" data-download-href="https://discuss.pytorch.org/uploads/default/090ab5c9d3ad7cb1cedd56e227fa496e30e85ec4" title="Screenshot from 2024-05-31 14-51-48"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/9/090ab5c9d3ad7cb1cedd56e227fa496e30e85ec4.png" alt="Screenshot from 2024-05-31 14-51-48" data-base62-sha1="1hZf6aAqB9HcNPylB271M9bI9ZG" width="690" height="80" data-dominant-color="303236"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2024-05-31 14-51-48</span><span class="informations">819×95 22.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Before initializing dpp I divide all the file paths in 8 parts. So is not each gpu supposed to work with only 20/8=2.5gb data?</p>

wilhelm: <p>Forget dpp.<br>
Try please with <em>accelerate</em></p>
Iterable dataloader threading errror Error

Tlotlo_Oepeng: <pre data-code-wrap="python"><code class="lang-python">
import math

class MyIterableDataset(torch.utils.data.IterableDataset):
    def __init__(self, start, end):
        super(MyIterableDataset).__init__()
        assert end &gt; start, "this example code only works with end &gt;= start"
        self.start = start
        self.end = end
    def __iter__(self):
        worker_info = torch.utils.data.get_worker_info()
        if worker_info is None:
            print("worker_info is none")  # single-process data loading, return the full iterator
            iter_start = self.start
            iter_end = self.end
        else:  # in a worker process
            print("worker_info is something:", worker_info) 
            # split workload
            per_worker = int(math.ceil((self.end - self.start) / float(worker_info.num_workers)))
            worker_id = worker_info.id
            iter_start = self.start + worker_id * per_worker
            iter_end = min(iter_start + per_worker, self.end)
        return iter(range(iter_start, iter_end))
# should give same set of data as range(3, 7), i.e., [3, 4, 5, 6].
ds = MyIterableDataset(start=3, end=7)

# Single-process loading
print(list(torch.utils.data.DataLoader(ds,)))

</code></pre>
<p>i  get error: /usr/lib/python3.10/multiprocessing/popen_fork.py:66: RuntimeWarning:</p>
<p>os.fork() was called. os.fork() is incompatible with multithreaded code, and JAX is multithreaded, so this will likely lead to a deadlock.</p>
<p>This is something im experiencing with iterable dataset when num_workers&gt;0 how do i fix this? and its slower for more workers?</p>

ptrblck: <p>You might want to remove Jax code from your PyTorch script assuming you are not using it.</p>

Tlotlo_Oepeng: <p>there is no Jax code I borrowed  this code from iterable torch doc’s i don’t know what causes this</p>

ptrblck: <p>The warning seems to be raised from <a href="https://github.com/google/jax/blob/d9f07d0350ad8f280a2415d126e6d4728414c698/jax/_src/xla_bridge.py#L128-L134"><code>jax/_src/xla_bridge.py</code></a>, so you might want to check which line of code raises it or where the import might be.</p>

Tlotlo_Oepeng: <p>i’ve tried searching to no avail would you recon maybe its python and i should try to update python? even in the code as you can see there is no jax, i think this is an issue with the iterable dataset</p>

ptrblck: <p>No, I doubt the error is raised by Python itself, as it would be unaware of Jax.<br>
Your code is missing the <code>import torch</code> statement, and after adding it I do not see any warning on my system but only this output:</p>
<pre data-code-wrap="python"><code class="lang-python">worker_info is none
[tensor([3]), tensor([4]), tensor([5]), tensor([6])]
</code></pre>
<p>After installing Jax and adding:</p>
<pre data-code-wrap="python"><code class="lang-python">import jax
jax._src.xla_bridge.backends()
</code></pre>
<p>I see the same warning.</p>

Tlotlo_Oepeng: <p>you sir are a genius!!, I unistalled jax and it worked. I had jax pre-installed in my  machine learning VM.</p>
FasterRCNN Mismatch: RuntimeError: Given groups=1, weight of size [256, 256, 3, 3], expected input[1, 512, 64, 64] to have 256 channels, but got 512 channels instead

morepenguins: <p>Hi!</p>
<p>I’m working on a customized Faster RCNN to be used with 4 channel 512x512 images.</p>
<p>I’m running into a size mismatch when I try to run my data through, and I’m not sure why.</p>
<p>Here’s my code:</p>
<pre data-code-wrap="python"><code class="lang-python">class CustomBackbone(torch.nn.Module):
    def __init__(self, backbone, out_channels):
        super().__init__()
        self.body = backbone
        self.out_channels = out_channels

    def forward(self, x):
        return self.body(x)

class ModifiedResNet50(resnet.ResNet):
'''
Modify ResNet to accept more channels and 512x512 images
'''
    def __init__(self, num_channels=4, pretrained=False):
        super().__init__(resnet.Bottleneck, [3, 4, 6, 3])
        
        self.conv1 = torch.nn.Conv2d(num_channels, 64, kernel_size=7, stride=2, padding=3, bias=False)
        
        if not pretrained:
            self._init_weights()

    def _init_weights(self):
        for m in self.modules():
            if isinstance(m, torch.nn.Conv2d):
                torch.nn.init.kaiming_normal_(m.weight, mode='fan_out', nonlinearity='relu')
            elif isinstance(m, torch.nn.BatchNorm2d):
                torch.nn.init.constant_(m.weight, 1)
                torch.nn.init.constant_(m.bias, 0)

class ModifiedFeaturePyramidNetwork(feature_pyramid_network.FeaturePyramidNetwork):
    def __init__(self, in_channels_list, out_channels, extra_blocks=None):
        super().__init__(in_channels_list, out_channels, extra_blocks)
        
        # Modify the lateral layers to match the backbone output channels
        self.lateral_layers = torch.nn.ModuleList([
            torch.nn.Conv2d(in_channels, out_channels, 1) for in_channels in in_channels_list
        ])

resnet50 = ModifiedResNet50(num_channels=4)
backbone_body = torchvision.models._utils.IntermediateLayerGetter(resnet50, return_layers=return_layers)
backbone = CustomBackbone(backbone_body, out_channels)

fpn = ModifiedFeaturePyramidNetwork(
    in_channels_list=in_channels_list,
    out_channels=out_channels,
    extra_blocks=None,
)

model = FasterRCNN(backbone, num_classes=n_classes, rpn_anchor_generator=AnchorGenerator())

trfrm = torchvision.models.detection.transform.GeneralizedRCNNTransform(min_size=300, max_size=1000, 
                                                                        image_mean=[0,0,0,0], image_std=[1,1,1,1],
                                                                        num_channels=n_channels)
model.transform = trfrm
</code></pre>
<p>Thanks in advance for any help with this!</p>
Train torchvision faster-rcnn on 4-channels images

Sterben: <p>Hello!<br>
I train Faster-RCNN from torchvision on 4-channels image. I changed resnet50 backbone for 4-channels input but I get error in “torchvision/models/detection/transform.py”.</p>
<pre><code class="lang-auto">return (image - mean[:, None, None]) / std[:, None, None]
RuntimeError: The size of tensor a (4) must match the size of tensor b (3) at non-singleton dimension 0
</code></pre>
<p>I can somehow turn off these transforms because right now I can’t train the model with them?<br>
Thanks for answers!</p>

AllenChen: <p>Did you fix this issue? If so, could you share how to skip this error? Thanks</p>

morepenguins: <p>I got the same error. It’s because PyTorch’s <code>FasterRCNN</code> has a <code>GeneralizedRCNNTransform</code> under the hood. I was able to fix the error by running the following:</p>
<pre data-code-wrap="python"><code class="lang-python">from torchvision.models.detection.transform import GeneralizedRCNNTransform
from torchvision.models.detection import FasterRCNN

n_channels=4

class ModTransform(GeneralizedRCNNTransform):
    def __init__(self):
        #adjust image means and image stds as needed
        #both should be the length of channels
        super().__init__(min_size=800, max_size=1300, 
                         image_mean=list(np.zeros(n_channels, int)), 
                         image_std=list(np.ones(n_channels, int)))

    def normalize(self, image):
        # Add your normalization needs
        return image

    def resize(self, image, target):
        # Add your resize needs
        return image, target

#Define FasterRCNN model according to your specs
model = FasterRCNN(...)

# Change the model transform to ModTransform
transform = ModTransform()
model.transform = transform
</code></pre>
ONNX interpreter using PyTorch as a backend

vadimkantorov: <p>Does anybody know of a ONNX interpreter using PyTorch as backend:</p>
<ul>
<li>e.g. this could be very naive interpreting graph nodes iteratively</li>
<li>or it could also do some graph transformation from ONNX graph to FX graph or sth like that</li>
<li>or it could consume ONNX graph and produce some sort of Python code (similar to FX graphs / ExportedProgram) executable and eager-debuggable with Python/PyTorch</li>
</ul>
<p>All these modes are useful in practice for educational purposes (debugging/understanding)  and also as a baseline (e.g. in context of tensorrt/polygraphy which benchmarks ORT versus TRT, but could also benchmark against some sorts of baseline PyTorch-powered interpreters) - especially if the ONNX file is coming from some third party / exporter.</p>
<p>If I understand correctly, at some point Caffe2 was filling this job, but I’m curious  if anything like this exists for modern PyTorch</p>
Cumulative sum with a discount factor

jiyuechenxing: <p>I’m looking for a algorithm to calculate cumulative sum with decay factor for a tensor. With the formula:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/1/51f1e33ce1574fea58a25e909176de2da42aa26b.png" alt="image" data-base62-sha1="bGUUQX81MS0NgKoDiHD4NiFZJW3" width="360" height="202"><br>
I’ve tried several methods, but all them make some errors:</p>
<ol>
<li><a href="https://github.com/toshas/torch-discounted-cumsum/tree/master" class="inline-onebox" rel="noopener nofollow ugc">GitHub - toshas/torch-discounted-cumsum: Fast Discounted Cumulative Sums in PyTorch</a>         This library seems to work but causes a series of errors about CUDA version.</li>
<li><a href="https://discuss.pytorch.org/t/cumulative-sum-with-decay-factor/69788/6">Cumulative sum with decay factor - PyTorch Forums</a>         Algorithm here has precision issues and results in nan value while calculating loss.</li>
<li>Inplace operations causes error while using autograd.</li>
</ol>
<p>Many thanks for your reply and help : D.</p>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/jiyuechenxing">@jiyuechenxing</a>,</p>
<p>Couldn’t you just compute the discount factor as an array and then element-wise multiplication it to the <code>x</code>? Then use <code>torch.cumsum</code> as usual?</p>
<p>Also, are the indicies correct with your equation? If <code>j=i</code>  then surely <code>\gamma^{j-1}</code> equals 1 by definition?</p>

jiyuechenxing: <aside class="quote no-group" data-username="AlphaBetaGamma96" data-post="2" data-topic="203801">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3da27b/48.png" class="avatar"> AlphaBetaGamma96:</div>
<blockquote>
<p>Also, are the indicies correct with your equation? If <code>j=i</code> then surely <code>\gamma^{j-1}</code> equals 1 by definition?</p>
</blockquote>
</aside>
<p>Yes, that’s the case.<br>
Now I realize the function with the method in <a href="https://discuss.pytorch.org/t/cumulative-sum-with-decay-factor/69788/6">Cumulative sum with decay factor - PyTorch Forums</a>. I add  small bias (1e-6) to the denominator to avoid Tensor/0. It indeed works but now I just don’t know how it would affect my model.<br>
Thank you again for your reply. <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face.png?v=12" title=":smiling_face:" class="emoji" alt=":smiling_face:" loading="lazy" width="20" height="20"></p>

AlphaBetaGamma96: <p>If you want to skip over a given element to avoid a divide-by-zero error, you could always just wrap the previous comment in a <code>torch.where</code> statement.</p>
<p>So, if the denominator is zero you just multiple by 1 and else use the array value as normal. This should remove the <code>1e-6</code> bias that you added to your calculation.</p>
NLP: Named Entity Recognition: Location: looking for a model supporting composite city names like like "Paris, TX"

new-in-town: <p>Some city names include the state/province/country to disambiguate cities. Examples:</p>
<pre><code class="lang-auto">   Paris, TX
   Moscow, ID
   Syracuse, NY
   Athens, United States
   Perth, GB
   Waterloo, Canada
</code></pre>
<p>Now there are some models capable to extract locations (so-called NER models). I tried these huggingface models (and few others):</p>
<pre><code class="lang-auto">   Davlan/distilbert-base-multilingual-cased-ner-hrl
   dslim/bert-base-NER
   FacebookAI/xlm-roberta-large-finetuned-conll03-english
</code></pre>
<p>None of them can handle such composite city names.<br>
They return one location for “Los Angeles” and “Frankfurt am Main”, but they return TWO separate locations for cities like Paris, TX.</p>
<p>So… who knows a pre-trained model supporting composite city names?</p>
Slow inference on HuggingFace timm models

Tanmoy_Mukherjee: <p>Hi folks<br>
I am trying to do inference on Imagenete with retrained resnet50 models using an MPS background (Apple Mac M3 Pro). I am noticing some operations like batch_norm are taking bulk of operation. Adding screenshots for the same<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/6/669fbb055a4b448300e8018cf80263525f492fae.png" data-download-href="https://discuss.pytorch.org/uploads/default/669fbb055a4b448300e8018cf80263525f492fae" title="Screenshot 2024-05-31 at 12.11.49"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/669fbb055a4b448300e8018cf80263525f492fae_2_690x130.png" alt="Screenshot 2024-05-31 at 12.11.49" data-base62-sha1="eDQVernSTo9U66yh0E4bwpIhfEG" width="690" height="130" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/669fbb055a4b448300e8018cf80263525f492fae_2_690x130.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/669fbb055a4b448300e8018cf80263525f492fae_2_1035x195.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/6/669fbb055a4b448300e8018cf80263525f492fae_2_1380x260.png 2x" data-dominant-color="282828"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2024-05-31 at 12.11.49</span><span class="informations">2070×392 38.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any insight will be appreciated to reduce this.</p>