akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Torch.cuda.is_available return True, but torch.randn(1).cuda() stuck

ch3nku1: <p>I’m new to cuda and pytorch, I ran a virtual machine via kubevirt and passed through a H100 for it. I installed nvidia-driver, anaconda and pytorch in the vm. Then I got this issue, as the screenshot. How can I fix this?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4d212dc9608a1d1e98326c83c3db307b8021ebc4.png" data-download-href="https://discuss.pytorch.org/uploads/default/4d212dc9608a1d1e98326c83c3db307b8021ebc4" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d212dc9608a1d1e98326c83c3db307b8021ebc4_2_537x500.png" alt="image" data-base62-sha1="b0jQJFLqmvTZQWlhvbg4imKWizO" width="537" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d212dc9608a1d1e98326c83c3db307b8021ebc4_2_537x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d212dc9608a1d1e98326c83c3db307b8021ebc4_2_805x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4d212dc9608a1d1e98326c83c3db307b8021ebc4_2_1074x1000.png 2x" data-dominant-color="080808"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1756×1634 249 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">apt install nvidia-driver-550
conda install -y pytorch torchvision torchaudio pytorch-cuda=12.4 -c pytorch -c nvidia
</code></pre>
<p><a href="https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-kubevirt.html#" class="onebox" target="_blank" rel="noopener nofollow ugc">https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-operator-kubevirt.html#</a></p>
Bivariate Timeseries for Neural Network

JosephSav: <p>I’m building a MLP to interpolate data from a timeseries .csv file. The problem is that my data looks something like this:<br>
t        x        y<br>
0.1    3        -<br>
0.2    -         5<br>
0.3    3.1      -<br>
0.4    -          -</p>
<p>In other words, I don’t have measurements for both x and y in most rows.<br>
Is there a way for me to use this data without having to use something like pandas.interpolate()?<br>
Here is some relevant code:</p>
<p>How I’m currently using the data:</p>
<pre><code class="lang-auto">class TimeSeriesDataset(torch.utils.data.Dataset):
    def __init__(self, csv_file):
        df = pd.read_csv(csv_file)

        df.replace('-', pd.NA, inplace=True)

        df['x'] = pd.to_numeric(df['x'])
        df['y'] = pd.to_numeric(df['y'])


        df['t'] = pd.to_timedelta(df['t'], unit='s')
        df.set_index('t', inplace=True)
        df = df.interpolate(method='time')
        df.reset_index(inplace=True)
        df['t'] = df['t'].dt.total_seconds()
        df.dropna(inplace=True)

        self.data = df

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        row = self.data.iloc[idx]
        x = torch.tensor([row['x']], dtype=torch.float32)
        y = torch.tensor([row['y']], dtype=torch.float32)
        t = torch.tensor([row['t']], dtype=torch.float32)

        labels = torch.tensor([x.item(), y.item()], dtype=torch.float32)
        return t, labels
</code></pre>
<p>training loop:</p>
<pre><code class="lang-auto">BATCH_SIZE = 50
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    print(device)

    dataset = TimeSeriesDataset(r'/content/data.csv')
    trainloader = torch.utils.data.DataLoader(
        dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=2)

    def loss_fn(outputs, labels):
        loss = torch.mean((labels - outputs) ** 2)
        return loss

    net = Net().to(device)

    optimizer = optim.Adam(net.parameters(), lr=0.001)

    for epoch in range(300):
        running_loss = 0.0
        for i, data in enumerate(trainloader, 0):
            inputs, labels = data
            inputs, labels = inputs.to(device), labels.to(device)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward + backward + optimize
            outputs = net(inputs)
            loss = loss_fn(outputs, labels)
            loss.backward()
            optimizer.step()

            # print statistics
            running_loss += loss.item()
            if i % 20 == 19:    # print every 20 mini-batches
                print('[%d, %5d] loss: %.3f' %
                    (epoch + 1, i + 1, running_loss / 20))
                running_loss = 0.0

    print('Finished Training')
</code></pre>
<p>I know that I could seperate the two variables and train them seperatly, I’m curious if there is any way I could keep it such that my model takes in one input ‘time’, and predicts the value for both variables ‘x’ and ‘y’.<br>
Cheers</p>
LibTorch with NVIDIA PyG Container/C++ within training loop

Wesxdz: <p>Hi, I’m doing some novel research involving integrating PyG with flecs entity component system.<br>
I am currently trying to train a model to learn the heuristic for A* pathfinding. The pathfinding code is written in C++. I am trying to understand how to use the total nodes in the open set as a loss parameter.<br>
I am trying to compile the latest LibTorch on <a href="https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pyg" class="inline-onebox" rel="noopener nofollow ugc">PyG | NVIDIA NGC</a><br>
via a Modal volume. <a href="https://modal.com/docs/guide/volumes" class="inline-onebox" rel="noopener nofollow ugc">Volumes | Modal Docs</a></p>
<p>Currently there is an error with loading the shared libraries.</p>
<blockquote>
<p>/example-app: symbol lookup error: ./example-app: undefined symbol: _ZN3c106detail23torchInternalAssertFailEPKcS2<br>
_jS2_RKSs</p>
</blockquote>
<p>Currently, I am investigating. But I would love help on either matching the right LibTorch version to this container, or otherwise understanding what is the best route for calling custom C++ within the training loop (especially for continual learning models).</p>

Wesxdz: <p>I was able to get LibTorch compiling by using a CUDA version matching the distro:<br>
<a href="https://download.pytorch.org/libtorch/cu124/libtorch-cxx11-abi-shared-with-deps-2.4.0%2Bcu124.zip" class="onebox" target="_blank" rel="noopener nofollow ugc">https://download.pytorch.org/libtorch/cu124/libtorch-cxx11-abi-shared-with-deps-2.4.0%2Bcu124.zip</a><br>
In terms of the C++ loss, I am considering a custom C++ PyTorch operator<br>
<a href="https://pytorch.org/tutorials/advanced/custom_ops_landing_page.html#custom-ops-landing-page" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Custom Operators — PyTorch Tutorials 2.4.0+cu121 documentation</a> to run the training code, and TorchScript for production.</p>
One of the differentiated Tensors does not require grad

Neda: <p>I’m trying to get a hessian vector product, but the following codes from <a href="https://github.com/quark0/darts/blob/master/cnn/architect.py" rel="nofollow noopener">DARTS</a> model don’t work as expected while <code>self.named_parameters()</code> is iterating over pairs <code>n,p.</code></p>
<p>the error is: <code>RuntimeError: One of the differentiated Tensors does not require grad</code></p>
<p>How can I fix this error? Thank you in advance.</p>
<pre><code class="lang-auto"> def _hessian_vector_product(self, vector, input, target, r=1e-2):
    R = r / _concat(vector).norm()
    for p, v in zip(self.model.parameters(), vector):
      p.data.add_(R, v)
    loss = self.model._loss(input, target)
    grads_p = torch.autograd.grad(loss, self.model.arch_parameters())

    for p, v in zip(self.model.parameters(), vector):
      p.data.sub_(2*R, v)
    loss = self.model._loss(input, target)
    grads_n = torch.autograd.grad(loss, self.model.arch_parameters())

    for p, v in zip(self.model.parameters(), vector):
      p.data.add_(R, v)

    return [(x-y).div_(2*R) for x, y in zip(grads_p, grads_n)]
</code></pre>

albanD: <p>Hi,</p>
<p>This comes from the fact that you call autograd.grad on something that does not require gradient.<br>
You should check where that happens and track down why this Tensor does not require gradients if it should.</p>

LLlearner: <p>I am having a similar problem. I am using vmap get the forward output for different ensembles and then taking the loss as a mean over all ensembles. When I run run autograd.grad on the loss I get the same error. The requires_grad flag of the loss is true. Any help in this matter?</p>

damkow: <p>Hello <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
You should add following lines before grad_p definition:<br>
for p in self.model.arch_parameters():<br>
p.requires_grad = True</p>
Cuda Graphs + index_copy_

Sam_Shleifer1: <p>I have an error I really dont understand when I do index_copy_ inside cuda graphs</p>
<pre><code class="lang-auto">[rank0]:     dst_BM.index_copy_(1, idx, src_BL)
[rank0]: RuntimeError: CUDA error: operation not permitted when stream is capturing
</code></pre>
<ul>
<li>this op suceeds for every layer in the first forward pass, and fails on the 2nd forward pass.</li>
<li>this op succeeds for a smaller model with different shapes on as many forward passes as you want. But there is no RAM pressure, using 42/80Gb.</li>
<li>this op is fine without cuda graphs</li>
<li>replacing with a triton kernel works inside cuda graphs, every thing is fine.</li>
<li>Strides are the same on forward pass 1 and 2</li>
<li>its impossible to reproduce offline</li>
</ul>
<p>Full Traceback: <a href="https://gist.github.com/sshleifer/f8531bc0b9bb576a274913af35bd0bc2" class="inline-onebox" rel="noopener nofollow ugc">index_copy_traceback.log · GitHub</a></p>
<p>Any ideas?</p>

ptrblck: <aside class="quote no-group" data-username="Sam_Shleifer1" data-post="1" data-topic="206992">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sam_shleifer1/48/61931_2.png" class="avatar"> Sam_Shleifer1:</div>
<blockquote>
<p>its impossible to reproduce offline</p>
</blockquote>
</aside>
<p>Does this mean you are unable to reproduce the failure using <code>index_copy_</code> alone?</p>

Sam_Shleifer1: <p>yes, index_copy_ no cuda graphs is fine</p>

ptrblck: <p>I’m still unsure I understand your previous description. Are you able to reproduce the issue in a standalone code snippet? If so, could you post it, please?</p>
Problem with auto_grad()

JosephSav: <p>Hi,<br>
I’m currently trying to implement a simple SGD without using a built in optimizer.<br>
What I’m running into is that after the first mini-batch, my loss loses its requires_grad status, and thus throws an error.<br>
Here is my code:</p>
<pre><code class="lang-auto">import numpy as np
import torch

def generate_data():
  data = torch.rand(1000, 2)
  label = ((data[:,0]+0.3*data[:,1]) &gt; 0.5).to(torch.int)
  return data[:,0], label

input, label = generate_data()

# Make minibatches.
inputs = torch.split(input, 32)
labels = torch.split(label, 32)

# Define the two variables to optimize
b1 = torch.autograd.Variable(torch.tensor([0.01]), requires_grad=True)
b2 = torch.autograd.Variable(torch.tensor([0.01]), requires_grad=True)

for epoch in range(15):
  for x, y in zip(inputs,labels):
    
    b1.grad = None
    b2.grad = None

    # Calculate p_x as per formula above
    p_x = 1 / (1 + torch.exp(-(b1 + b2 * x)))

    # Calculate the negative loss likelihood
    l = (y * torch.log(p_x) + (1 - y) * torch.log(1 - p_x)).sum()

    l.backward()
    # Calculate the gradient of the loss w.r.t. the inputs
    delta_b1 = b1.grad
    delta_b2 = b2.grad
    # Update the parameters b according to SGD formula

    with torch.no_grad():
      b1 = b1 - 0.01 * delta_b1
      b2 = b2 - 0.01 * delta_b2

</code></pre>
<p>P.S. I am aware Variable is deprecated, but I am working around given code.<br>
Any help would be greatly appreciated.</p>

ptrblck: <p>You are replacing the trainable and deprecated <code>Variable</code>s inside the <code>no_grad</code> block with constant tensors. You could try to update these variables via inplace methods instead.</p>

JosephSav: <p>I see. Is it acceptable to change to tensor.data directly? Like so:</p>
<pre><code class="lang-auto">b1.data -= 0.1 * delta_b1
    b2.data -= 0.1 * delta_b2
</code></pre>
<p>Otherwise I get:<br>
RuntimeError: a leaf Variable that requires grad is being used in an in-place operation.</p>

ptrblck: <p>Since you are already using deprecated <code>Variable</code> objects you could also use the internal <code>.data</code> attribute, but note that I would strongly recommend updating your code and to use supported approaches.</p>

JosephSav: <p>Ok, i really appreciate the help!</p>
Classification of image temporally

sidb23: <p>Hi, I have a problem where in, I want to classify images into class A, B, C based on the objects in the image. I also want to introduce a temporal vector which stores the time elapsed from the start of the sequence. I want to use this as well to predict the class A, B, and C which are dependent on time as well.<br>
I understand how they can be trained as separate tasks, but I am curious to know can they be combined as a single task and trained?</p>

Daniel_Gonzalez1: <p>Hi, I think It’s essential to consider whether you need to relate changes across a sequence. If you want to use both the image itself and a time value from the start as inputs to your model, you can treat the time value as an additional input feature. However, if you aim to capture temporal dependencies, consider encoding a window of frames. This makes your model ‘temporal aware’ and is it used for video-related tasks.</p>
Help with installing detectron2

Yash1: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/8/08397bb28c364c785a391f52c94c3fbb5df61318.png" data-download-href="https://discuss.pytorch.org/uploads/default/08397bb28c364c785a391f52c94c3fbb5df61318" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/8/08397bb28c364c785a391f52c94c3fbb5df61318.png" alt="image" data-base62-sha1="1aKYzlqTgOlwXzysixBFoYqy2De" width="690" height="271" data-dominant-color="0F0F0F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1095×431 13.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I am having problem installing torch and torchvision in conda environment could someone suggest a compatible version for cuda 12.1 driver</p>

Daniel_Gonzalez1: <p>Hi, you can try installing <code>cudatoolkit</code> package in the conda enviroment, pytorch should use the one in the environment instead, and conda will make sure the version is compatible with the rest of the packages installed. You can add the nvidia channel as well with <code>-c nvidia</code> so that conda can find more versions.</p>
What's are the out argument uses?

Manik_Hossain: <p><code>torch.</code> <code>stack</code> ( <em>seq</em> ,  <em>dim=0</em> ,  <em>out=None</em> ) → Tensor<br>
here what 's the use cases of out arguments ?</p>

ptrblck: <p>It’s just an optional argument to store the output to.<br>
E.g. if you already have the tensor initialized, you could pass it to the function:</p>
<pre><code class="lang-python">out = torch.empty(2, 10)
# ...
torch.stack((torch.randn(10), torch.randn(10)), dim=0, out=out)
</code></pre>

justinliu: <p>Sorry to bump up a very old thread. I assume the point of using the <code>out</code> argument when you already have an initialized tensor is to have a slight performance advantage over not setting <code>out</code>, is that right? Or are there other considerations?</p>
Retrun data type of torch.fx.tracer.trace()

Ardeal: <p>Hi,<br>
according to the instruction here: <a href="https://pytorch.org/docs/main/fx.html#torch.fx.Tracer.trace" class="inline-onebox" rel="noopener nofollow ugc">torch.fx — PyTorch main documentation</a><br>
the return data of <code>torch.fx.tracer.trace()</code> and <code>torch.fx.symbolic_trace()</code> is <code>graph</code>.<br>
however, the data type of <code>fx_model</code> is <code>GraphModule</code> in the following code:</p>
<pre data-code-wrap="python"><code class="lang-python">    fx_model = torch.fx.symbolic_trace(model)
    modules = dict(fx_model.named_modules())
    for node in fx_model.graph.nodes:
        if node.target in modules:
            print(node.target)
            if type(modules[node.target]) == torch.nn.Conv2d:
                print('conv node')
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38bbf6f11f0fde9a55510af7561b8f81df373bd3.png" data-download-href="https://discuss.pytorch.org/uploads/default/38bbf6f11f0fde9a55510af7561b8f81df373bd3" title="1722602928748"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/8/38bbf6f11f0fde9a55510af7561b8f81df373bd3.png" alt="1722602928748" data-base62-sha1="85TrWpsYUhlL6XKaJ1iU0GNW871" width="690" height="467" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1722602928748</span><span class="informations">804×545 36.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Issue with torchvision: torchvision::nms does not exist

kr17bakrebs: <p>I am having issues with torchvision. I am working with the following NVIDIA driver:<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2024 NVIDIA Corporation<br>
Built on Wed_Apr_17_19:19:55_PDT_2024<br>
Cuda compilation tools, release 12.5, V12.5.40<br>
Build cuda_12.5.r12.5/compiler.34177558_0</p>
<p>when running this test script:</p>
<p>import torch<br>
import torchvision<br>
print(“CUDA available:”, torch.cuda.is_available())<br>
print(torchvision.<strong>version</strong>)</p>
<p>i get the following error:</p>
<p>Traceback (most recent call last):<br>
File “/runningTest.py”, line 2, in <br>
import torchvision<br>
File “/lib/python3.9/site-packages/torchvision/<strong>init</strong>.py”, line 6, in <br>
from torchvision import _meta_registrations, datasets, io, models, ops, transforms, utils<br>
File “/lib/python3.9/site-packages/torchvision/_meta_registrations.py”, line 164, in <br>
def meta_nms(dets, scores, iou_threshold):<br>
File “/lib/python3.9/site-packages/torch/library.py”, line 467, in inner<br>
handle = entry.abstract_impl.register(func_to_register, source)<br>
File “/lib/python3.9/site-packages/torch/_library/abstract_impl.py”, line 30, in register<br>
if torch._C._dispatch_has_kernel_for_dispatch_key(self.qualname, “Meta”):<br>
RuntimeError: operator torchvision::nms does not exist</p>
<p>can someone help with this? it was working fine until i accidentally downgraded to Pytorch 2.3 earlier, i have since tried to download 2.4, which did not work and went back to 2.3. It is quite urgent <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

ptrblck: <aside class="quote no-group" data-username="kr17bakrebs" data-post="1" data-topic="207411">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/5fc32e/48.png" class="avatar"> kr17bakrebs:</div>
<blockquote>
<p>it was working fine until i accidentally downgraded to Pytorch 2.3 earlier,</p>
</blockquote>
</aside>
<p>The issue might be raised by a mismatching installation between PyTorch and <code>torchvision</code> based on your description. What kind of error do you get if you want to reinstall the latest PyTorch release?</p>
How to use custom collate_fn by update defalut_collate_fn_map

chener: <p><a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.default_collate" rel="noopener nofollow ugc">torch.utils.data — PyTorch 2.3 documentation</a></p>
<p>In the page, it says <code>default_collate_fn_map.update({torch.Tensor: collate_tensor_fn})</code> can be used to update defalut_collate. For my case, I want to add <code>default_collate_fn_map.update({PIL.Image: lambda x: return x})</code>. However, I cannot change default_collate_fn_map because it’s defined in torch/utils/data/_utils/collate.py. Any help?</p>

Albert_Ou: <pre data-code-wrap="python"><code class="lang-python">import torch
torch.utils.data._utils.collate.default_collate_fn_map.update({})
</code></pre>
<p>Seems the <code>default_collate_fn_map</code> is located in a internal package _utils.<br>
It’s not exposed to user as the <code>default_collate</code> api.</p>
Is torchvision also available in C++?

Shisho_Sama: <p>I was wondering if torchvision is also available in C++. especially the transforms, and utils modules.</p>

LeviViana: <p>Yep -&gt; <a href="https://github.com/pytorch/vision#c-api" rel="nofollow noopener">https://github.com/pytorch/vision#c-api</a></p>
<p>Moreover, most of torchvision’s transforms use torch functionals as primitives, and these are available through libtorch.</p>

Shisho_Sama: <p>Thanks, but I couldnt find any transform related thing  there! Only Image encode/decore were added plus nms and roi, which is not what I’m after. Do you mean I have to reimplement everything myself using the libtorch functions?</p>

LeviViana: <p>What transforms are you interested in ? If you need something like <a href="https://github.com/pytorch/vision/blob/master/torchvision/transforms/transforms.py#L692" rel="nofollow noopener">this</a>, I think yes you’ll need to reimplement it, is that right <a class="mention" href="/u/fmassa">@fmassa</a>?</p>

Shisho_Sama: <p>Yes, I guess I’ll be implementing a lot of stuff already!<br>
Thanks a lot really appreciate your time and kind response.<br>
Have a great day</p>

Shisho_Sama: <p>Heres an update of what I recently found. it might come in handy for people like me that later on see this thread.<br>
Last night I noticed some functionalities in the torchvision has been moved to <strong>torch::data</strong> namespace in the C++ API, specially the <strong>transform</strong> that I was looking for: so check  <strong>torch::data::transforms::</strong> out as well</p>
<p>Its still very crude, but it offers some good necessaities such as <code>BatchTransform</code>, <code>Normalize</code>, <code>Transform</code>, etc</p>

teacherli07: <p>Hello, I’m trying to convert my python files to c++, mostly with the help of github copilot.<br>
i have something in my python code like this:</p>
<pre><code class="lang-auto">    test_transforms = transforms.Compose([transforms.Resize(224), transforms.CenterCrop(224), transforms.ToTensor(),
                                         transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])
</code></pre>
<p>the Copillot rewrite it in c++ to:</p>
<pre><code class="lang-auto">    auto test_transforms = torch::data::transforms::Compose({
        torch::data::transforms::Resize(224),
        torch::data::transforms::CenterCrop(224),
        torch::data::transforms::ToTensor(),
        torch::data::transforms::Normalize({0.485, 0.456, 0.406}, {0.229, 0.224, 0.225})
    });
</code></pre>
<p>however, there seems no torch::data::transforms::Compose or others, except for torch::data::transforms::Normalize, which was found in the namespace.</p>
<p>i installed libtorch and torchvision, and i’m wondering whether transforms::Compose is available, or i shall rewrite transforms.Compose into c++ by myself?<br>
thanks a lot</p>
Conv2D producing a bunch of negative values from 0-1 data

rankinstudio: <p>I’m having some strange issues with conv2D.</p>
<p>I have single layer images I have normalized from 0-1 (verified this is done correctly)<br>
When I run them through convolution I am getting a bunch of negative values.</p>
<pre><code class="lang-auto">in_channel = 1 #Grayscale
kernel_size = 5
out_channels = 7

conv1 = nn.Conv2d(in_channel, out_channels, kernel_size)

plot_imgs(xs.detach().numpy(), "Original")

x = conv1(xs)
plot_imgs(x.detach().numpy(), "Convolution 1")
</code></pre>
<p>Input is normalized 0-1, output has a bunch of negative values (verified).<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/e/4e6bc2e4301802d866239e77113d02b8b18ba832.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4e6bc2e4301802d866239e77113d02b8b18ba832" title="stacked"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4e6bc2e4301802d866239e77113d02b8b18ba832_2_279x500.jpeg" alt="stacked" data-base62-sha1="bbK7s3Q6Xnyyrijg0XqEMa3hZse" width="279" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4e6bc2e4301802d866239e77113d02b8b18ba832_2_279x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4e6bc2e4301802d866239e77113d02b8b18ba832_2_418x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/e/4e6bc2e4301802d866239e77113d02b8b18ba832_2_558x1000.jpeg 2x" data-dominant-color="808080"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">stacked</span><span class="informations">994×1776 348 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have verified there are negatives in the output images. I have tried different kernel and output channels.</p>
<p>Any idea how to avoid this?</p>
<p>Adding torch.backends.cudnn.deterministic = True didn’t seem to change anything.</p>
<p>Thanks!</p>

Soumya_Kundu: <p>I think the random filters/bias applied by conv2d contains negative values.</p>
<p><a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html</a></p>

rankinstudio: <p>It seems to be randomly inverting the images. I’m not sure that is going to help the model.</p>
Find the topk over masked input

Eli_Sason: <p>Hi, I have an input tensor X with dimensions (B, H,L,L).<br>
I also have a binary mask M with same dimension. Each row may have diffenret number of active values.</p>
<p>I would like to get the topk values over the last dim, only over the masked value!.<br>
The issue is that i want to be efficent, and avoid from making unesseriy operations.</p>
<p>Can someone help?<br>
Thanks!</p>

Soumya_Kundu: <p>Is this what you are looking for:</p>
<pre><code class="lang-auto">import torch

def masked_topk(X, M, k):
    B, H, L, _ = X.shape
    
    masked_X = X * M    
    masked_X = masked_X.masked_fill(~M.bool(), float('-inf'))
    
    topk_values, topk_indices = torch.topk(masked_X, k=k, dim=-1)
    
    valid_mask = topk_values != float('-inf')
    
    return topk_values, topk_indices, valid_mask

#Run:
B, H, L = 2, 3, 5
k = 3

X = torch.rand(B, H, L, L)
M = torch.randint(0, 2, (B, H, L, L)).float() #Mask

topk_values, topk_indices, valid_mask = masked_topk(X, M, k)
</code></pre>

Eli_Sason: <p>hi,</p>
<p>thanks on reply, but no … It is a bit more complex issue <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"><br>
Maybe i was not very clear in the problem definition.</p>
<p>-This topk mode is not efficeint, although it will give you the right solution…<br>
It will give you the same complexity as working without the mask.</p>
<p>Ideally I looking for solution where the complexity of the topk will be influence with the mask!<br>
“torch.topk(input, k, mask, dim)”<br>
It should be much faster to find the topk with a handful of inputs (just the active) instead of looking all over the entire dimension.</p>
<p>note:<br>
-I have considered to re-write the torch topk, but it seems to be big task. It has also cpp + cuda code. and I did not find documentation.</p>
<p>Thanks!</p>
Majority voting

MERYEM: <p>Can I use majority voting with softmax activation function outputs in PyTorch to aggregate predictions from a group of classifiers, like 4 CNN models, by combining their softmax probabilities? Additionally, how would approaches like hard, soft, and weighted voting be applied in this context?</p>

Soumya_Kundu: <p>You could do it like this:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F

# Suppose you have 4 models and their softmax outputs for a batch of inputs
softmax_outputs = [model(input_data) for model in models]

# Convert softmax outputs to class predictions
class_predictions = [torch.argmax(output, dim=1) for output in softmax_outputs]

# Stack the predictions and compute the mode
stacked_predictions = torch.stack(class_predictions, dim=0)
majority_vote_predictions, _ = torch.mode(stacked_predictions, dim=0)

</code></pre>
<p>Nice bit on making things faster with vmap: <a href="https://pytorch.org/tutorials/intermediate/ensembling.html" class="inline-onebox" rel="noopener nofollow ugc">Model ensembling — PyTorch Tutorials 2.4.0+cu121 documentation</a></p>

MERYEM: <p>Could you please provide a detailed explanation of the steps involved in applying majority voting? Additionally, could you clarify whether this process is considered hard, soft, or weighted voting?</p>

Soumya_Kundu: <p>Majority voting, from my understanding, is hard voting.</p>
<p>For a detailed analysis on voting in general, you could refer to this: <a href="https://machinelearningmastery.com/voting-ensembles-with-python/" rel="noopener nofollow ugc">https://machinelearningmastery.com/voting-ensembles-with-python/</a></p>

MERYEM: <p>I can’t find the mathematical equation for each one</p>

Soumya_Kundu: <p><a href="https://rasbt.github.io/mlxtend/user_guide/classifier/EnsembleVoteClassifier/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://rasbt.github.io/mlxtend/user_guide/classifier/EnsembleVoteClassifier/</a></p>
Why is a tensor non-contiguous on one machine, but contiguous on another?

ricsi046: <p>I’m running the exact same code on 2 virtual machines, each of them has the same os, same python package versions.</p>
<pre><code class="lang-auto">from multimodal_transformers.model import BertWithTabular, TabularConfig
from transformers import BertTokenizer, BertConfig

if __name__ == "__main__":
    pretrained_model_name = "SZTAKI-HLT/hubert-base-cc"
    tokenizer = BertTokenizer.from_pretrained(pretrained_model_name)
    bert_config = BertConfig.from_pretrained(pretrained_model_name)
    tabular_config = TabularConfig(
        combine_feat_method="text_only",
        cat_feat_dim=0,
        numerical_feat_dim=0,
        num_labels=7,
    )
    bert_config.tabular_config = tabular_config

    model = BertWithTabular.from_pretrained(pretrained_model_name, config=bert_config)
</code></pre>
<p>When I check model.state_dict(), I can see some of the tensors are always non-contiguous on one machine, while all of them are contiguous on the other machine, which results in error when trying to save the model. I can’t figure out what determines if the tensor is contiguous or not. Do I need to convert tensors to contiguous before saving? I don’t feel like that is normal.</p>
<p>I would also like to mention that the first machine didn’t have this problem a few weeks ago, no idea what changed.</p>

ptrblck: <p>This is indeed strange. Does the error show which tensors are not contiguous? If so could you check how these tensors were initialized and possibly manipulated in your code?</p>

ricsi046: <p>They come from a pretrained model (<a href="https://huggingface.co/SZTAKI-HLT/hubert-base-cc" class="inline-onebox" rel="noopener nofollow ugc">SZTAKI-HLT/hubert-base-cc · Hugging Face</a>), so I don’t know how the tensors were initialized and manipulated, but the problem is definitely related to the model, since I tried loading another model (<a href="https://huggingface.co/google-bert/bert-base-uncased" class="inline-onebox" rel="noopener nofollow ugc">google-bert/bert-base-uncased · Hugging Face</a>), and it doesn’t have this problem, all the tensors are contiguous.</p>
<p>So I guess there is nothing we can do about it.</p>
Is there any way to get traced GraphModule without creating torch dynamo's constraint guards?

gabriel4321: <p>Hi, I want to trace my model and convert it into GraphModule. And this GraphModule will be provided to our compiler. For this, I’m using <code>torch._dynamo.export</code>,  but it takes huge amount of  time for tracing my model, and it found out a lot of time is spent on creating and updating guards which is not necessary for my case. It must be much faster if I can disable guard update and creation, is there any way to do so? Or, is there some kind of method to reduce tracing time?</p>
Module name updated after saving

Laya1: <p>I trained a Faster RCNN model and evaluated it with <code>torchmetrics</code>. I had saved a dictionary with model weights in <code>.pt</code> format along with the <code>torchmetrics.detection.map</code> object after training an object detection model. I now upgraded torchmetrics and the module name has changed from <code>map</code> to <code>mean_ap</code>. Now when I use <code>torch.load</code>, I get the error <code>ModuleNotFoundError: No module named 'torchmetrics.detection.map'</code>. How do I solve this without having to downgrade <code>torchmetrics</code>? Thanks in advance for your help.</p>

ptrblck: <p>I assume you have saved the <code>torchmetrics</code> object directly, which might require the same source code during the load process. I don’t know if this object contains a <code>state_dict</code> but if so you should save and restore it instead, as the actual data will be saved only instead of the object.</p>
DataLoader CPU bottleneck with on-the-fly data on GPU

buswayne: <p>Hi everyone!</p>
<p>I’m training a model with synthetic data that i’m generating on-the-fly with an IterableDataset. I made sure the data generation runs entirely on the GPU, since I want to reduce the CPU usage to the bare minimum.</p>
<p>The issue is that, when I’m training the model, i’ve noticed with the profiler that there is still a bottleneck on the CPU when loading the data with the DataLoader. I’m confused because I’m sure the tensors that I’m generating are on the GPU (i cannot set pin_memory=True for example cause they already are on GPU), but at the same time if i change the num_workers parameter performance are affected (from what I understand, the workers should be relevant just when using the CPU).</p>
<p>Do you have any suggestion to solve this issue?</p>
<p>Thank you in advance</p>

ptrblck: <aside class="quote no-group" data-username="buswayne" data-post="1" data-topic="207118">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/buswayne/48/71761_2.png" class="avatar"> buswayne:</div>
<blockquote>
<p>at the same time if i change the num_workers parameter performance are affected (from what I understand, the workers should be relevant just when using the CPU).</p>
</blockquote>
</aside>
<p>Increasing the number of workers creates new processes corresponding to each worker which will load and process the data in the background and could thus also yield benefits in your use case.</p>

buswayne: <p>Hi ptrblck,</p>
<p>Thank you for the reply.</p>
<p>Actually I’ve fixed the issue switching from IterableDataset to a Dataset. The function enumerate(dataloader) with <strong>iter</strong> uses CPU but the “equivalent” next(item(dataloader)) with <strong>getitem</strong> does not and the code runs smoothly on the GPU. I don’t know why but issue solved <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> thanks again</p>
Best practices to keep track of augmentations pipelines

mdf: <p>Hi there,<br>
I’m working on a instance segmentation task where I make heavy use of data augmentation on-the-fly during training. I’m experimenting with different augmentation pipelines (in terms of transformations applied and params for those transformations) and I usually lose track of what augmentation I used pretty easily as I run more and more experiments.<br>
My question is, for sake of trackability/reproducibility, what are some best practices to keep track of these transformations?</p>
<p>So far I tried to have one function for each augmentation pipeline, but the number of functions easily explodes because of all the possible combinations I try.</p>

Eduardo_Lawson: <p>When i used to track my experiments like that i was using <a href="https://wandb.ai/" rel="noopener nofollow ugc">wandb</a>. I used to log the run in this api and put a note in the run for remember wich augmentations and pararemeters were used. You can use wand.init(notes:) for this: <a href="https://docs.wandb.ai/ref/python/init" rel="noopener nofollow ugc">DOC</a>. And you will be able to track the results aswell.</p>

mdf: <p>Yeah I’m doing something similar now by dumping the configuration in a config file so I can track and use it. I hoped for something more structured but this is working for now. Thank you for you feecback</p>
Accuracy between Softmax 's .backward() and calculated according to the formula?

czhxiaohuihui: <p>I try to calculate the grad of softmax like the following code:</p>
<pre><code class="lang-auto">def softmax_backward(y, grad_output):
    dtype = y.dtype
    y = y.to(torch.float64)
    grad_output = grad_output.to(torch.float64)
    y_grad_output = y * grad_output
    grad_input = y*(grad_output - torch.sum(y_grad_output, dim=-1, keepdim=True))
    return grad_input.to(dtype)
</code></pre>
<p>but i get some mismatch cases compared with .backward(), I want to know why?</p>
<p>Following is all of my code:</p>
<pre><code class="lang-auto">import torch
from copy import deepcopy

def softmax_backward(y, grad_output):
    dtype = y.dtype
    y = y.to(torch.float64)
    grad_output = grad_output.to(torch.float64)
    y_grad_output = y * grad_output
    grad_input = y*(grad_output - torch.sum(y_grad_output, dim=-1, keepdim=True))
    return grad_input.to(dtype)

mydtype = torch.bfloat16
a = torch.randn((2048,8), dtype=mydtype, requires_grad=True, device="cuda")
x = torch.tensor(a*3, dtype=mydtype, requires_grad=True, device="cuda")

y = x.softmax(dim=-1)
y_copy = deepcopy(y.detach())

y_grad = torch.tensor(torch.randn(y.shape, dtype=mydtype)*3, dtype=mydtype, device="cuda")
y_grad_copy = deepcopy(y_grad)

y.backward(y_grad)

x_grad = softmax_backward(y_copy, y_grad_copy)
atol = 0.001
rtol = 0.016

count = 0 
for i in range(a.size(0)*a.size(1)):
    xi = x_grad.reshape(-1)[i].item()
    xi_gt = x.grad.reshape(-1)[i].item()
    err = abs(xi-xi_gt)
    tol_err = (atol + rtol * abs(xi_gt))
    if err &gt; tol_err:
        count += 1
        print(f"a = {xi}, b={xi_gt}, err={err}, tol_err={tol_err}")

print("mismatch count= ", count)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c70b3f2a4f78fd07aa8077053ebb1588ea1fb3d.png" data-download-href="https://discuss.pytorch.org/uploads/default/6c70b3f2a4f78fd07aa8077053ebb1588ea1fb3d" title="图片"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/c/6c70b3f2a4f78fd07aa8077053ebb1588ea1fb3d.png" alt="图片" data-base62-sha1="ftj1XRQxdbXk2BIQxUAp0f33EN7" width="690" height="386" data-dominant-color="303030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">图片</span><span class="informations">1233×690 65 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Is deepcopying nn.Module objects still recommended?

shahbaz: <p>A <a href="https://github.com/pytorch/tutorials/issues/2177" rel="noopener nofollow ugc">github issue</a> on the Pytorch repo suggests againt using <code>deepcopy</code> to clone a <code>torch.nn.Module</code> model and the recommended way is to use <code>torch.save</code> instead followed by <code>torch.load</code></p>
<p>Could you clarify what exactly are the issues concerning the use of <code>deepcopy</code> and if it is strongly advised not to use it particularly if the model class has additional attributes registered as buffers ?</p>
Difference in training results depending on how often eval loop is run

Ludek: <p>Hi,</p>
<p>This is snippet of my loop, I either evaluate after every epoch or only on the last epoch. When I evaluate after every epoch, the per-epoch training loss i slightly different than if I evaluate only on last epoch → running evaluation changes train results.<br>
I have progresively striped down eval loop and the result is that it is sufficient that I loop through the val_loader to reproduce these changes. Random seed is fixed at the start of whole script (both torch and numpy). I have read on stack overflow that looping train_loader → val_loader → train_loader can influence shuffling if it is set to true, regardless of seed. But I get the same discrepencies even with shuffle=False in both loaders.</p>
<pre><code class="lang-auto">for e in epochs:
    model.train().cuda()
    for imgs, masks in train_loader:
        optimizer.zero_grad()
        imgs, masks = (
            imgs.cuda().float(),
            masks.cuda().float(),
        )
        with torch.cuda.amp.autocast():
            pred_mask = model(imgs)
            batch_train_seg_loss = loss_fn(pred_mask, masks)
  
        scaler.scale(batch_train_seg_loss).backward()
        scaler.step(optimizer)
        scaler.update()
        scheduler.step()

    if ((e + 1) % eval_freq) == 0 or (e == epochs - 1):
        with torch.no_grad():
            model = model.eval()
            for imgs, masks in progress_bar(val_loader, len(val_loader), parent=mb):
                 #here can be just pass or whole val loop, the traning discrepency behavior is the 
                 #same
    
</code></pre>
<p>Not that it has significant effect on the results, but it is unxplainable to me and the experiments are harder to control/explain.</p>
<p>Thank you for any clarification!</p>

ptrblck: <p>This might be expected due to <a href="https://discuss.pytorch.org/t/why-iterating-over-a-deterministic-dataloader-changes-the-seed-state/199610/3">this behavior</a>.</p>
Fp16 inference time cost

hexisyztem: <p>When using torch fp16 for inference, a significant decrease in the time consumption of the cuda kernel is observed through nvidia nsight profile. However, there is no change in the overall time consumption when testing end-to-end.<br>
By observing the profile information, it can be confirmed that the computational accuracy used in the inference process is indeed fp16 precision.</p>

hexisyztem: <p>related repo: <a href="https://github.com/FunAudioLLM/CosyVoice" class="inline-onebox" rel="noopener nofollow ugc">GitHub - FunAudioLLM/CosyVoice: Multi-lingual large voice generation model, providing inference, training and deployment full-stack ability.</a></p>

ptrblck: <p>Your use case might be CPU-limited and you can verify it by profiling it with a visual profiler (e.g. Nsight Systems). In this case, you could try to use CUDA Graphs as it will reduce the CPU overhead significantly.</p>
When calling torch.Tensor.to('cuda'), is the data loaded into shared GPU memory (SRAM) or global GPU memory (DRAM)?

gpgu: <p>Hi, perhaps this is a silly question, but what I could find so far is that <code>to('cuda')</code> returns a “CUDA Tensor” or puts it into “CUDA memory”, but I’m not sure where that is exactly on the chip (global GPU memory or shared of the thread block)?</p>
<p>Two hints that point to shared memory:</p>
<ul>
<li>From <code>torch.Tensor.is_shared()</code>: “[…] is always <code>True</code> for CUDA tensors.”</li>
<li>From <code>torch.Tensor.share_memory_()</code>: “This is a no-op […] for CUDA tensors.”</li>
</ul>
<p>Could someone confirm? So we’d never need to call <code>.share_memory_()</code> on a Tensor that was already moved <code>.to('cuda')</code>, correct? Thanks!</p>

ptrblck: <p>The tensor will be stored in the global memory of the GPU. The shared memory can be used in the CUDA kernels themselves.</p>

gpgu: <p>Thanks for the super quick answer <a class="mention" href="/u/ptrblck">@ptrblck</a> ! That is very clear. Do you know why <code>is_shared()</code> is always <code>True</code> then? Is it just a default because parts of the tensor might be in GPU shared memory (depending on kernels), but not tracked? And I’m guessing <code>.share_memory_()</code> is referring to host shared memory, and not GPU shared memory?</p>

ptrblck: <aside class="quote no-group" data-username="gpgu" data-post="3" data-topic="207333">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/41988e/48.png" class="avatar"> gpgu:</div>
<blockquote>
<p>And I’m guessing <code>.share_memory_()</code> is referring to host shared memory, and not GPU shared memory?</p>
</blockquote>
</aside>
<p>Yes. Since CUDA tensors do not need to be moved for sharing across processes, <code>is_shared</code> returns always <code>True</code> for tensors stored on the GPU.</p>
Custom QA model over fitting(Transformers)

Sushmita_Upadhyay1: <p>I am training a custom QA model and my validation accuracy is starting from 61% and training from 54% after that training accuracy is continuously increasing and training loss decreasing whereas validation accuracy only increases till 3rd epoch to reach 64% (and training accuracy at 3rd epoch is 71%) and then starts decreasing and validation loss increases. I have around 90000 data and my model has 18.9M training parameter. I suspect that it is over fitting. I have tried everything to fix this my nothing helped. Can anyone help to identify the issue. Also even if there is over fitting can I still use the result till the 3rd epoch as validation loss increases after that? I am using pytorch lightning to train</p>
Phoeneme level forced alignment w/ torch audio?

m172021: <p>I have a large collection of files of the form english_word.wav . Example: hello.wav, stop.wav, go.wav, one.wav, two.wav,  …</p>
<p>I would like to segment them into monophones / diphones.</p>
<p>I am aware of <a href="https://pytorch.org/audio/stable/tutorials/forced_alignment_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Forced Alignment with Wav2Vec2 — Torchaudio 2.4.0 documentation</a> – but it appears that only segments wav files into the WORD level – which I already have.</p>
<p>Is there a simple way via torchaudio to segment wav files t the phoeneme level ?</p>
<p>I am aware of the existence of <a href="http://montreal-forced-aligner.readthedocs.io/en/latest/" class="inline-onebox" rel="noopener nofollow ugc">Montreal Forced Aligner documentation — Montreal Forced Aligner 3.0.0 documentation</a> , /kaldi-asr.org/ , www.fon.hum.uva.nl/praat/</p>
<p>but I would prefer something with less dependencies and runs directly on torchaudio if possible.</p>
<p>Thanks!</p>
<p>Context: goal is to write a dumb simple monophone / diphone ‘concat synthesis’ text to speech; yes, I realize neural is all the hype and concat synthesis is out dated, but I’m just curious / want to see what it sounds like.</p>
<p>EDIT: got hit with ‘new user can only put 2 links in a post’</p>