akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Module 'torch.nn' has no attribute 'Module'

username: Wayne_127
cooked: <p>what shoud i do?<br>
my code is as follows:<br>
import torch<br>
import torch.nn as nn<br>
class LSTM(nn.Module):</p>
<pre><code>def __init__(self, num_classes, input_size, hidden_size, num_layers):
    super(LSTM, self).__init__()
    
    self.num_classes = num_classes
    self.num_layers = num_layers
    self.input_size = input_size
    self.hidden_size = hidden_size
    #self.seq_length = seq_length
    self.dropout = nn.Dropout(p=0.2)
    
    self.lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size,
                        num_layers=num_layers, batch_first=True,dropout = 0.25)
    
    self.fc = nn.Linear(hidden_size, num_classes)

def forward(self, x):
    h_0 = Variable(torch.zeros(
        self.num_layers, x.size(0), self.hidden_size).to(device))
    
    c_0 = Variable(torch.zeros(
        self.num_layers, x.size(0), self.hidden_size).to(device))
    
    # Propagate input through LSTM
    ula, (h_out, _) = self.lstm(x, (h_0, c_0))
    
    h_out = h_out.view(-1, self.hidden_size)
    
    out = self.fc(h_out)
    out = self.dropout(out)
   
    return out
</code></pre>
<hr>
<p>AttributeError                            Traceback (most recent call last)<br>
Cell In[32], line 3<br>
1 import torch<br>
2 import torch.nn as nn<br>
----&gt; 3 class LSTM(nn.Module):<br>
5     def <strong>init</strong>(self, num_classes, input_size, hidden_size, num_layers):<br>
6         super(LSTM, self).<strong>init</strong>()</p>
<p>AttributeError: module ‘torch.nn’ has no attribute ‘Module’</p>

username: jonathanjsmith
cooked: <p>You can try listing the attributes of the  <code>torch.nn</code> namespace with <code>torch.nn.__dict__</code>.</p>
<p><code>Module</code> should be defined in this namespace.</p>

username: Wayne_127
cooked: <p>thank u for ur advice, I reinstall the pytorch, now the problem has been solved, i guess it may be some mistakes when first setting the environment.</p>
How to implement Opacus in a multi-model setup?

username: Anirban_Nath
cooked: <p>In my work my model is split up into four components say <code>m1, m2, m3, m4</code>. The output of <code>m1</code> is sent through <code>m2</code>, which is then sent through <code>m3</code> and finally <code>m4</code> gives the output. All models share a common optimizer, which holds the weights of all models and updates weights for all models.</p>
<p>It seems I cannot individually call make_private on <code>m1, m2, m3 and m4</code> because I am getting some mismatched dimensionality error from module <code>m2</code> onwards.</p>
<pre><code class="lang-auto">self.m1, self.DPOptimizer, self.DPDataloader = self.privacy_engine.make_private(
            module=self.m1,
            optimizer=self.optimizer,
            data_loader=self.data_loader,
            noise_multiplier=1.0,  # sigma
            max_grad_norm=self.clip,
        )
self.m2, _, _ = self.privacy_engine.make_private(
            module=self.m2,
            optimizer=self.optimizer,
            data_loader=self.data_loader,
            noise_multiplier=1.0,  # sigma
            max_grad_norm=self.clip,
        )
self.m3, _, _ = self.privacy_engine.make_private(
    module=self.m3,
    optimizer=self.optimizer,
    data_loader=self.data_loader,
    noise_multiplier=1.0,  # sigma
    max_grad_norm=self.clip,
)
self.m4, _, _ = self.privacy_engine.make_private(
    module=self.m4,
    optimizer=self.optimizer,
    data_loader=self.data_loader,
    noise_multiplier=1.0,  # sigma
    max_grad_norm=self.clip,
)
</code></pre>
<p>In this setting what is the correct way to implement the make_private function?</p>

username: alexandresablayrolle
cooked: <p>Thanks for reaching out and sorry for the delayed response. Any reason why you can’t chain the modules in a <code>nn.Sequential</code>?</p>

username: Vaishnav
cooked: <p>I came across a similar issue. I’m building a synthetic data generation framework using attention. The final loss comes from multiple loss terms (each from different models), and I’m using a single optimiser to carry out the optimisation task. In this setting, how to use Opacus’s PrivacyEngine 's make_private function?</p>
Module 'torch' has no attribute 'rename_privateuse1_backend'

username: aohenuo
cooked: <p>my torch vision is 2.1.0</p>

username: jonathanjsmith
cooked: <p><code>rename_privateuse1_backend</code> is defined in the <code>torch.utils</code> namespace so using <code>torch.utils.rename_privateuse1_backend(backend_name)</code> should work.</p>

username: aohenuo
cooked: <p>thanks so much! this function solved my question</p>
Audio spectrogram data normalization

username: Isaac_Kargar
cooked: <p>I have a dataset of audio files and want to do data normalization on my data in spectrogram space. I read each file and apply data augmentation on the fly.</p>
<p>1- If these augmentations change the statistics, what should we do? What is the way it’s done in vision models? I see channel-wise mean and std which I think data augmentations do not change them, but masking or some others, I think, change it. Is it true?</p>
<p>2- Is normalizing different frequencies and times in the spectrogram using the same scale (instance-based, batch-based or globally) okay? What are the best practices there?</p>
<p>3- Should we normalize based on frequency for example or time?</p>
Change .pth to .ptl

username: lucas_Nardelli
cooked: <p>I have a .pth file, how can i change to .ptl?</p>

username: ptrblck
cooked: <p>Could you explain what these files contain? I assume the <code>.pth</code> file stored PyTorch <code>state_dict</code>s?</p>

username: lucas_Nardelli
cooked: <p>The .pth file stored PyTorch OrderedDict</p>

username: ptrblck
cooked: <p>In this case it seems the <code>.pth</code> file was created via <code>torch.save(model.state_dict(), "PATH.pth")</code>. What is the <code>.ptl</code> file supposed to contain? I assume it should differ in a way?</p>
Dataparallel only outputs one half of the total batch size on 2 gpus

username: Rgb1998
cooked: <p>Hi all,</p>
<p>When I run the following CONV-LSTM model with input of shape (20,1,4,128,128), where the batch is in dimension 0:</p>
<p><span class="hashtag-raw">#shape</span> (batch, time, channels, height, width)</p>
<p>device = torch.device(“cuda:0” if torch.cuda.is_available() else “cpu”)<br>
model= nn.DataParallel(model, device_ids=[0, 1])<br>
model = model.to(device)<br>
a = torch.randn(20,1,4,128,128).to(device)<br>
output,_ = model(initial_states, a)<br>
print(output.shape)</p>
<p>The output shape is (10, 2, 4, 128, 128) rather than (20, 2, 4, 128, 128). Any reason why this is the case?</p>
<p>Best<br>
Rami</p>

username: Rgb1998
cooked: <p>Hello,</p>
<p>I found the reason for the error: it’s because my forward method has multiple arguments (i.e. model.forward(self, x, h, c) rather than just model.forward(self, x) ). However I am using a conv-LSTM architecture so the other arguments h and c are required. What is the solution in this case?</p>
<p>cc <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>Best<br>
Rami</p>

username: ptrblck
cooked: <p>Note that <code>nn.DataParallel</code> is in maintenance mode and you should thus use <code>DistributedDataParallel</code> which should not suffer from these issues as each process uses its own input.</p>
Modifying the model checkpoint

username: riririririi
cooked: <p>Let’s say I have trained a resnet-50 model and i have replaced its last fc layer with a custom defined layer which performs l2 normalisation on the output of the avgpooling layer, making the last few layers look like this:<br>
-----previous layers------<br>
)<br>
(avgpool): AdaptiveAvgPool2d(output_size=(1, 1))<br>
(fc): Sequential(<br>
(0): L2Normalization()<br>
)<br>
)</p>
<p>i have saved the model as a checkpoint with the model’s state dictionary saved in it, now let’s say i want to add a few layers to perform feature reduction and i throw in batchnorm layers so that my model looks like something below:<br>
-----previous layers------<br>
)<br>
(avgpool): AdaptiveAvgPool2d(output_size=(1, 1))<br>
(fc): Sequential(<br>
(0): Linear(in_features=2048, out_features=1024, bias=True)<br>
(1): BatchNorm1d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)<br>
(2): Linear(in_features=1024, out_features=256, bias=True)<br>
(3): L2Normalization()<br>
)<br>
)</p>
<p>now i want to train this model on the same data, so it would be beneficial for me to load the weights from the previous model (for layers other than the ones i added later), but i wont be able to do so as there would be a layer mismatch.<br>
is there any way i could modify the checkpoint to have some random values for the extra layers that are not present in the checkpoint or is there anything else that i could do to load the values from the layers that are present in the model and randomly initialise values for the newly added layers ?<br>
i am new to pytorch and deep learning so i feel kinda lost here, feel free to let me know if i am doing anything incorrectly.</p>

username: ptrblck
cooked: <p>The simplest approach might be to load the <code>state_dict</code> into the model before adding the new layers. Alternatively, you could also add new key-value pairs into the <code>state_dict</code> as it’s derived from a <code>dict</code>.</p>
Compiling PyTorch 1.7.0 with CUDA compute capability 3.0 fails due to declaration conflict

username: s6toweis
cooked: <p>Hey, I’m trying to build PyTorch to work with a NVIDIA Quadro K4000.<br>
I set up CUDA 10.1.243 and tested it.<br>
Then I installed cuDNN 7.6.2.24.<br>
I found several tutorials in the internet that claimed that PyTorch 1.7.0 should work with CUDA 10.1.<br>
But I can’t use the legacy binaries since they only work with higher compute capability.<br>
A lot of posts in this forum claim that you can get newer versions of PyTorch to work with lower compute capability by installing them from source. So that is what I tried.</p>
<p>I followed the tutorial from the PyTorch 1.7.0 Readme: <a href="https://github.com/pytorch/pytorch/tree/v1.7.0" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch at v1.7.0</a><br>
I installed all common dependencies and magma-cuda101.<br>
And at first, everything works and my computer compiles the first ~3000 files without problems.<br>
(I’m using GCC 8.3.0.)</p>
<p>But then it fails like this:</p>
<pre><code class="lang-auto">In file included from /nosave/pytorch/aten/src/ATen/cuda/nvrtc_stub/ATenNVRTC.cpp:1:
/nosave/pytorch/aten/src/ATen/cuda/nvrtc_stub/ATenNVRTC.h:33:5: error: declaration of ‘nvrtcResult (* at::cuda::NVRTC::nvrtcVersion)(int*, int*)’ [-fpermissive]
   _(nvrtcVersion)                                \
     ^~~~~~~~~~~~
/nosave/pytorch/aten/src/ATen/cuda/nvrtc_stub/ATenNVRTC.h:101:45: note: in definition of macro ‘CREATE_MEMBER’
 #define CREATE_MEMBER(name) decltype(&amp;name) name;
                                             ^~~~
/nosave/pytorch/aten/src/ATen/cuda/nvrtc_stub/ATenNVRTC.h:102:3: note: in expansion of macro ‘AT_FORALL_NVRTC’
   AT_FORALL_NVRTC(CREATE_MEMBER)
   ^~~~~~~~~~~~~~~
In file included from /nosave/pytorch/aten/src/ATen/cuda/nvrtc_stub/ATenNVRTC.h:5,
                 from /nosave/pytorch/aten/src/ATen/cuda/nvrtc_stub/ATenNVRTC.cpp:1:
/software/easybuild/software/CUDA/10.1.243/include/nvrtc.h:92:13: error: changes meaning of ‘nvrtcVersion’ from ‘nvrtcResult nvrtcVersion(int*, int*)’ [-fpermissive]
 nvrtcResult nvrtcVersion(int *major, int *minor);
</code></pre>
<p>Surprisingly, I could not find a single other case of this error on Google.</p>
<p>I’ve tried this with export TORCH_CUDA_ARCH_LIST=3.0 and without it.</p>
<p>This looks like a compatibility issue for me, but I’m not sure how that is possible.<br>
I know that PyTorch 1.7.0 should work with CUDA 10.1…</p>
<p>Can somebody please help me compile this?<br>
Or give me an alternative approach that allows me to accelerate learning on this GPU?</p>
How to give additional parameters to the collate_fn passed in data loader?

username: mahesh_bhosale
cooked: <p>I want to use a custom collate function which collates data based on the arguments passed to it. I am not sure if it’s possible to pass the arguments to collate function internally from the data loader.</p>

username: ptrblck
cooked: <p>I don’t know where this argument would be coming from, but in case you want to set an additional argument in your <code>collate_fn</code> while creating the <code>DataLoader</code>, you could use a <code>lambda</code> approach.<br>
If you want to pass this argument from the <code>Dataset</code>, you could return it and use it in your custom <code>collate_fn</code>.</p>
<p>Here is a simple approach showing these approaches:</p>
<pre><code class="lang-python"># simple use case
def my_collate(batch):
    x = torch.stack([a for a, b in batch])
    y = torch.stack([b for a, b in batch])
    return x, y

dataset = TensorDataset(torch.arange(10), torch.arange(10, 20))
loader = DataLoader(dataset, batch_size=5, collate_fn=my_collate)

for data, target in loader:
    print(data, target)


# with argument
def my_collate(batch, arg):
    print(arg)
    x = torch.stack([a for a, b in batch])
    y = torch.stack([b for a, b in batch])
    return x, y

loader = DataLoader(dataset, batch_size=5, collate_fn=lambda batch: my_collate(batch, arg="myarg"))

for data, target in loader:
    print(data, target)
</code></pre>

username: pbelcak
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="154738">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>lambda batch: </code></p>
</blockquote>
</aside>
<p>This sadly won’t work for DDP, since lambdas can’t be pickled. Any workarounds?</p>

username: pbelcak
cooked: <p>So, I gather there are no workarounds. Everything that needs to be passed to the <code>collate_fn</code> must be explicitly included in the dataset.</p>
<p>The good news is that this actually relatively memory-cheap to do. Just make sure that you are not making copies of whatever you need to have at hand in <code>collate_fn</code> in your dataset implementation – share the objects.</p>
Torch 2.1.0 with different cuda versions

username: Gennaro_Vaccaro
cooked: <p>Hi.<br>
I’m trying to install torch version 2.1.0 but with cuda 11.7.</p>
<p>I tried this installation: python3.9 -m pip install --no-cache-dir torch --index-url <a href="https://download.pytorch.org/whl/cu117" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu117</a><br>
But strangely, it installs torch 2.1.0+cu121.</p>
<p>On my Pod there is the installation of nvidia 515.65.01+ cudnn 8.5.0.96, with cuda 11.7, and by installing torch 2.1.0+cu121, torch is no longer able to use the GPU.</p>
<p>Trying to install only torch==2.0.1 instead (python3.9 -m pip install torch==2.0.1), torch works.</p>
<p>Is Torch version 2.1.0 only compatible with Cuda 12.1? Is there a way to use it with cuda 11.7?<br>
Thank you</p>

username: ptrblck
cooked: <p>If you don’t want to update your NVIDIA driver making it compatible with CUDA 12.x, you could install the PyTorch binaries with CUDA 11.8 as given in the install instructions <a href="https://pytorch.org/get-started/locally/">here</a>.</p>

username: Gennaro_Vaccaro
cooked: <p>Thanks. Installing torch 2.1.0 with cuda 11.8 seems to work with a cuda 11.7 installation on pod.</p>
Old PyTorch version for new GPU

username: viborgen
cooked: <p>Hi,<br>
I’m currently trying to install a library which requires torch==1.5. I’m using a Nvidia RTX 4090 GPU and when I run nvidia-smi: cuda is 12.2<br>
nvcc --version: release 11.5</p>
<p>However, when I try to run Torch with the library I get:<br>
RuntimeError: CUDA error: no kernel image is available for execution on the device</p>
<p>Is there anything I can do?</p>

username: ptrblck
cooked: <p>Cou could try to cherry-pick all needed commits for CUDA updates into your local PyTorch 1.5 version and build it from source afterwards. However, it sounds cumbersome and could easily fail if you are missing some important changes so I would recommend updating your script instead to be compatible with the latest PyTorch release.</p>

username: viborgen
cooked: <p>Thank you, will try that.</p>
<p>I am also having some troubles using it with Torch==1.13.1. Is my GPU not supported in this version, or would it help if I try to downgrade Cuda?</p>
<p>Thank you for your help.</p>

username: ptrblck
cooked: <p>It depends which PyTorch binary you are installing as you would need CUDA 11.x and the binaries must be built with sm_86+.</p>
NVIDIA GeForce RTX 3090 Ti with CUDA capability sm_86 is not compatible with the current PyTorch installation. The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70

username: nkkaushal
cooked: <p>Hi</p>
<p>I am running my code which uses torch but I am getting this error</p>
<p>“<strong>NVIDIA GeForce RTX 3090 Ti with CUDA capability sm_86 is not compatible with the current PyTorch installation. The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70</strong>”</p>
<p>I have CUDA version 12.2 (checked using nvidia-smi)</p>
<p>nvcc -V :<br>
NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2021 NVIDIA Corporation<br>
Built on Thu_Nov_18_09:45:30_PST_2021<br>
Cuda compilation tools, release 11.5, V11.5.119<br>
Build cuda_11.5.r11.5/compiler.30672275_0</p>
<p>torch version : ‘1.10.1+cu102’. (using torch.<strong>version</strong>)</p>
<p>I want to use the same torch version, is there a way to resolve this issue??</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Same as <a href="https://discuss.pytorch.org/t/help-with-gaussian-splatting-torch-4090-torch-version-2-1-0/189924/2">here</a>. Your locally installed CUDA toolkit won’t be used and you need to install a PyTorch binary with CUDA&gt;=11.x.</p>

username: nkkaushal
cooked: <p>But I want to use same version of Pytorch, because if I change my Pytorch version then I have to change my code at multiple locations and I want to avoid that. Is there any other fix for this??</p>

username: nkkaushal
cooked: <p>Thanks a lot, its resolved</p>

username: jack_lee
cooked: <p>hi，how do you resolve this problem? I get the same question.</p>

username: ptrblck
cooked: <p>Install any current stable or nightly release as all binaries support up to sm_90 now.</p>
Save the gradient of the intermediate tensor

username: wangchengtao
cooked: <p>I have a question, if I want to save the gradient of the intermediate tensor generated in backward for future use, what should I do? Below is a small demo. Thanks!</p>
<pre><code class="lang-python">import torch
import gc
a = []
class Exp(torch.autograd.Function):
    @staticmethod
    def forward(ctx, i):
        result = torch.exp(i)
        ctx.save_for_backward(result)
        return result
    @staticmethod
    def backward(ctx, grad_output):
        result, = ctx.saved_tensors
        a.append(grad_output.detach())
        return grad_output * result
exp = Exp()
x = torch.tensor([3., 4.], requires_grad=True)
y = exp.apply(x)
y.sum().backward()
gc.collect()
torch.cuda.empty_cache()
print(a)
</code></pre>

username: wangchengtao
cooked: <p>cc <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
A similar question to <a href="https://discuss.pytorch.org/t/use-custom-stream-in-custom-torch-autograd-function/189996">Use custom stream in custom torch.autograd.Function</a></p>

username: soulitzer
cooked: <p>Your custom Function idea sounds fine to me. You can also register a backward hook to <code>y</code> here, e.g. <code>y.register_hook(fn)</code> and do the same appending idea.</p>
<p>Alternatively, if you were okay until the end of backward to use this gradient, you can just specify it as part of the <code>inputs=</code> you pass to <code>.grad()/.backward()</code>, e.g. `y.sum().backward(inputs=(y, […])), this would mean however that you’d need to explicitly pass the rest of things you want to compute grad wrt to into inputs=.</p>
Feature extraction for similarity learning

username: riririririi
cooked: <p>i am training a model to perform feature extraction on images and then i am performing clustering on those features to group the images based on visual similarity.<br>
i am using a resnet-50 model, removed the final classification layer and added a custom L2 normalisation layer, the output of the L2 normalisation layer is what i am using to perform clustering on, the final layers look like this:<br>
-----previous layers-----<br>
)<br>
(avgpool): AdaptiveAvgPool2d(output_size=(1, 1))<br>
(fc): Sequential(<br>
(0): L2Normalization()<br>
)<br>
)</p>
<p>i want to know if i should add a linear layer or multiple linear layers inside a sequential container or should i not use any linear layers at all ?<br>
could someone please comment on where could i take the output from or should i avoid or use linear layers and/or activation functions ?</p>
Use custom stream in custom torch.autograd.Function

username: wangchengtao
cooked: <p>I want to know if there are any notes for using custom stream in custom torch.autograd.Function. My custom Function is as follows. I want to offload input and grad_output to CPU side. Is the code wrong?</p>
<pre><code class="lang-python">import torch
from torch.nn import functional as F

class LinearFunction(torch.autograd.Function):
    copy_stream = torch.cuda.Stream(priority=-1)
    tensors = []
    @staticmethod
    def pack_to_cpu(tensor):
        packed = torch.empty(
            tensor.size(),
            dtype=tensor.dtype,
            layout=tensor.layout,
            pin_memory=True)
        packed.copy_(tensor, non_blocking=True)
        return (tensor.device, packed)

    @staticmethod
    def unpack_from_cpu(packed):
        device, tensor = packed
        return tensor.to(device, non_blocking=True)

    @staticmethod
    def forward(ctx, input, weight, bias=None):
        ctx.save_for_backward(input, weight, bias)
        output = F.linear(input, weight, bias)
        return output

    @staticmethod
    def backward(ctx, grad_output):
        input, weight, bias = ctx.saved_tensors
        grad_input = grad_weight = grad_bias = None
        if ctx.needs_input_grad[0]:
            grad_input = grad_output @ weight
        if ctx.needs_input_grad[1]:
            # grad_weight = grad_output.t().mm(input)
            grad_weight = grad_output.view(-1, grad_output.size()[-1]).t() @ input.view(-1, input.size()[-1])
        if bias is not None and ctx.needs_input_grad[2]:
            assert False, "bias is not supported yet!"
            grad_bias = grad_output.sum(0).squeeze(0)
        with torch.cuda.stream(LinearFunction.copy_stream):
            input.record_stream(LinearFunction.copy_stream)
            grad_output.record_stream(LinearFunction.copy_stream)
            input_detach = input.detach()
            grad_output_detach = grad_output.detach()
            input_cpu = LinearFunction.pack_to_cpu(input_detach)
            grad_output_cpu = LinearFunction.pack_to_cpu(grad_output_detach)
            LinearFunction.tensors.append((input_cpu, grad_output_cpu))
        return grad_input, grad_weight, grad_bias
</code></pre>

username: wangchengtao
cooked: <p>cc <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Could you help me?</p>
Rescaling the Generated Output to the pixel intensity range in DCGAN

username: mini870312
cooked: <p>I have a question about the <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html#dcgan-tutorial" rel="noopener nofollow ugc">DCGAN tutorial</a></p>
<p>I was wondering why the Generator does not rescale the generated output to the range [0,1] or [0, 255] (the range of pixel intensity) for each channel before sending it to the Discriminator during the training process.</p>
<p>I thought that if the Generator model did not rescale the generated output, the Discriminator could easily learn the input images that did not include the value in the range [0,1] or [0, 255] as the fake image.</p>
Loss function defined by myself does not match with CrossEntropyLoss()

username: Benja
cooked: <p>Hi!. I defined a loss function (mleLoss; I have attached a snapshot of it) to be equivalent to CrossEntropyLoss() (the reason of this is not important right now), but during training I get loss=nan. I thought that maybe this was because at some point the loss function takes one 0 and because we are doing -log(p_i) if p_i=0 then the loss function diverges. However when a see the outputs (softmax probabilities) when loss=nan the maximum probabilities correspond to their labels, so I do not see why it happens loss=nan. Furthermore, when I use CrossEntropyLoss() as my loss function I do not get loss=nan, what leads me to think that the problem is in the definition of the loss function mleLoss(), but to me my loss function and CrossEntropyLoss() are identical and I do not see where is the error. Thanks in advance!!</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bfdd7265b6ff7403100ae96f6f3688633a98113d.png" alt="mylossfunction" data-base62-sha1="rnjDXKfI491d9a5dwWSQhYrFPdz" width="630" height="89"></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/0/30932ff0bbeafce89ff82ba087a8fe7931e01ff7.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/30932ff0bbeafce89ff82ba087a8fe7931e01ff7" title="code_snippet"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30932ff0bbeafce89ff82ba087a8fe7931e01ff7_2_690x457.jpeg" alt="code_snippet" data-base62-sha1="6VIgtUKBfXtjlSj6zB6x0Vaekl1" width="690" height="457" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30932ff0bbeafce89ff82ba087a8fe7931e01ff7_2_690x457.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30932ff0bbeafce89ff82ba087a8fe7931e01ff7_2_1035x685.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/0/30932ff0bbeafce89ff82ba087a8fe7931e01ff7_2_1380x914.jpeg 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">code_snippet</span><span class="informations">1411×936 138 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
How do we interpret a multivariate forecasting model with Captum?

username: zezimabig
cooked: <p>Dear pytorch community,</p>
<p>I am currently faced with an issue of not being able to carry out the evaluation of the importance of features of my multivariate forecasting model.</p>
<p>Here is the model architecture:</p>
<pre><code class="lang-auto">class MultivariateLSTM(nn.Module):
    
    def __init__(self, input_size, hidden_size, num_layers, output_size):
        super(MultivariateLSTM, self).__init__()
        self.hidden_size = hidden_size
        self.lstm = nn.LSTM(input_size = input_size, hidden_size = hidden_size, num_layers = num_layers, batch_first = True, dropout = 0.2)
        self.fc1 = nn.Linear(hidden_size, 8)
        self.fc2 = nn.Linear(8, output_size)
        self.relu = nn.ReLU()
        # self.relu = SmeLU(6)
        
    def forward(self, x):
        
        out, _ = self.lstm(x)
        out = self.relu(out)
        out = self.fc1(out[:, -1, :])
        out = self.relu(out)
        out = self.fc2(out)
        
        return out
</code></pre>
<p>Here are the sample shapes used as inputs and outputs:</p>
<pre><code class="lang-auto">train_X_tensor.shape, train_y_tensor.shape = torch.Size([10339, 12, 3]) torch.Size([10339, 6])
</code></pre>
<p>Note that the structure of the input follows the (samples, seq_len, features) convention used in forecasting models.</p>
<p>Here is the code block used to evaluate the feature importance:</p>
<pre><code class="lang-auto">ig = IntegratedGradients(model)
ig_nt = NoiseTunnel(ig)
dl = DeepLift(model)
gs = GradientShap(model)
fa = FeatureAblation(model)

ig_attr_test = ig.attribute(test_X_tensor, n_steps=50)
ig_nt_attr_test = ig_nt.attribute(test_X_tensor)
dl_attr_test = dl.attribute(test_X_tensor)
gs_attr_test = gs.attribute(test_X_tensor, train_X_tensor)
fa_attr_test = fa.attribute(test_X_tensor)
</code></pre>
<p>and this is the current error:</p>
<pre><code class="lang-auto">----&gt; 7 ig_attr_test = ig.attribute(test_X_tensor, n_steps=50)
AssertionError: Target not provided when necessary, cannot take gradient with respect to multiple outputs.
</code></pre>
<p>Does anyone have experience evaluating with Captum? I greatly appreciate if you could share your thoughts and experiences in this matter!</p>
<p>Thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Debug FSDP program

username: 111357
cooked: <p>I am trying to debug the fsdp program, but it does not support setting breakpoints with pdb.<br>
Can anyone share some experience in debugging the fsdp program?</p>

username: wconstab
cooked: <p>I guess the issue you’re running into is one of pdb being incompatible with multiprocessing.  There are some workarounds you can try.  One is to configure stdin and then use pdb.  See this tutorial: <a href="https://pubhub.devnetcloud.com/media/pyats/docs/aetest/debugging.html" class="inline-onebox" rel="noopener nofollow ugc">Debugging Resources — pyATS Documentation</a></p>
<p>another option is to use rpdb, which requires opening a port and telnet connecting to it for debugging.</p>
<p>In either case, you’ll have to be careful about only setting the breakpoint on one particular rank instead of on all ranks</p>

username: 111357
cooked: <p>Thanks very much. The below methods work for me.</p>
<pre><code class="lang-auto">import sys
import pdb

class ForkedPdb(pdb.Pdb):
    """A Pdb subclass that may be used
    from a forked multiprocessing child

    """
    def interaction(self, *args, **kwargs):
        _stdin = sys.stdin
        try:
            sys.stdin = open('/dev/stdin')
            pdb.Pdb.interaction(self, *args, **kwargs)
        finally:
            sys.stdin = _stdin


...
ForkedPdb.set_trace()
...
</code></pre>
Why does LBFGS stop before reaching the max_iter?

username: bowen_shen
cooked: <p>Hello, when training a neural network using LBFGS as the optimization algorithm, I’ve set the maximum number of iterations to 5000 and very small tolerance values, like 1e-222. However, sometimes it stops after just a few iterations, and other times it exceeds the maximum number of iterations. How can I address this issue?</p>

username: tom
cooked: <p>Did you check whether it sees an exact 0 in the gradients?</p>
<p>Best regards</p>
<p>Thomas</p>
Gradient clipping and gradient accumulation together

username: loveis98
cooked: <p>I try to do gradient accumulation (from <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#gradient-accumulation" rel="noopener nofollow ugc">this</a>) for RNN model, so I also need to clip gradient. Could you please clarify if I’m doing rigth these two operations in the code below (in the right order and logic)?</p>
<pre><code class="lang-auto">loss_function = nn.BCEWithLogitsLoss()
num_batches = 1
running_loss = 0.0
iters_to_accumulate = 4
scaler = GradScaler()

model.train()
for batch in tqdm(train_generator, desc='Training'):
        with autocast(device_type=device.type, dtype=torch.float16):
            output = torch.flatten(model(batch['features'], batch['category'])).to(device)
            batch_loss = loss_function(output, batch['label'].float())
            batch_loss = batch_loss / iters_to_accumulate

        scaler.scale(batch_loss).backward()

        if (num_batches) % iters_to_accumulate == 0:
            scaler.unscale_(optimizer)
            torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.)
            scaler.step(optimizer)
            scaler.update()
            optimizer.zero_grad()

        num_batches += 1
</code></pre>

username: soulitzer
cooked: <p>When you do gradient accumulation are you supposed to average the gradients first before running the optimizer? If the point of gradient accumulation is to amortize the cost of the optimizer over a greater number of steps, then I’d guess no.</p>
<p>In that case, you should increase your max_norm corresponding to how many iters you accumulate. Another question is how you are deciding your max_norm in the first place.</p>
<p>That’s just my intuition, but curious what the literature says here.</p>

username: loveis98
cooked: <p>Thank you, I didn’t find any evidence-based articles on how to choose max_norm value, I chose it empirically, depending on how the gradient norm of my model changes</p>

username: loveis98
cooked: <p>I would like to read the opinion of moderators also:)</p>

username: ptrblck
cooked: <p>Your code looks fine as it sticks to <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#gradient-accumulation">this example</a>.</p>
Gradient of input and model parameters

username: Saizor
cooked: <p>Hello,</p>
<p>I have a question regarding training a model’s parameters when the forward function include a gradient wrt to the input. Instead of showing all the code (where I have trouble with the backward-function), I can boil it down to this small (and silly) example (and when I can solve this problem I think it will translate to my code):</p>
<p>import torch as th<br>
import torch.nn as nn<br>
import torch.nn.functional as F</p>
<p>class Model(nn.Module):</p>
<pre><code>def __init__(self):
    super().__init__()
    self.l1 = nn.Linear(5, 2)

def forward(self, x):
    x_ = self.l1(x)
    x_ = (x_ ** 2).sum(1)
    return th.autograd.grad(x_, x, grad_outputs=th.ones_like(x_))[0]
</code></pre>
<p>m = Model()<br>
m.train()<br>
x = th.randn(10, 5).requires_grad_(True)<br>
y = m(x)</p>
<p>loss = F.mse_loss(y, x)<br>
params = m.parameters()<br>
for p in params:<br>
print(th.autograd.grad(loss, p, retain_graph=True)[0])</p>
<p>Here I cannot compute the gradient in the for-loop. How can I solve this?<br>
Thanks in advance!!</p>

username: KFrank
cooked: <p>Hi Saizor!</p>
<aside class="quote no-group quote-modified" data-username="Saizor" data-post="1" data-topic="190052" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/c2a13f/48.png" class="avatar"> Saizor:</div>
<blockquote>
<p>I have a question regarding training a model’s parameters when the forward function include a gradient<br>
…</p>
<pre><code>def forward(self, x):
    x_ = self.l1(x)
    x_ = (x_ ** 2).sum(1)
    return th.autograd.grad(x_, x, grad_outputs=th.ones_like(x_))[0]
</code></pre>
<p>y = m(x)</p>
<p>loss = F.mse_loss(y, x)<br>
params = m.parameters()<br>
for p in params:<br>
print(th.autograd.grad(loss, p, retain_graph=True)[0])</p>
<p>Here I cannot compute the gradient in the for-loop.</p>
</blockquote>
</aside>
<p>In the call to <code>autograd.grad()</code> in <code>Model.forward()</code> you need to use<br>
<code>create_graph = True</code>.</p>
<p>(This is the same issue as the second question in the earlier thread you<br>
posted.   Or was that thread from Josue?)</p>
<p>Best.</p>
<p>K. Frank</p>

username: Saizor
cooked: <p>Oh, thanks!</p>
<p>That must be someone else. Sorry, I did not find that thread!</p>
<p>Once again, thanks!</p>
<p>Best!</p>
Cache batches of weighted samples for faster yielding

username: mra-h
cooked: <p>Hello,</p>
<p>I have a dataset with 82 million samples. The samples can be categorize either as being inside or outside a mask. The number of inside pixels is much lower than the number of outside pixels. However, I would like to have a balanced distribution of in/out samples in every batch.</p>
<p>At the moment I am implementing a custom <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.WeightedRandomSampler" rel="noopener nofollow ugc">WeightedSampler</a> where my <code>__iter__()</code> method looks like the following. I am making it compatible with DDP as well:</p>
<pre><code class="lang-auto">    def __iter__(self):
        sample_size = self.sample_size * self.num_replicas
        num_in_mask = int(sample_size * self.weight)
        num_out_mask = sample_size - num_in_mask

        g = torch.Generator()
        if self.rank == 0:
            self.counter += 1
        g.manual_seed(self.seed + self.epoch + self.counter % torch.iinfo(int).max)

        in_mask_sample = torch.randint(len(self.dataset.in_mask), size=(num_in_mask,), generator=self.generator)
        out_mask_sample = torch.randint(len(self.dataset.out_mask), size=(num_out_mask,), generator=self.generator)
        in_mask_indices = self.dataset.in_mask[in_mask_sample]
        out_mask_indices = self.dataset.out_mask[out_mask_sample]
        indices = torch.cat((in_mask_indices, out_mask_indices))
        permuted = torch.randperm(len(indices), generator=self.generator)
        indices = indices[permuted]

        indices = indices[self.rank:len(indices):self.num_replicas]

        return iter(indices.tolist())
</code></pre>
<p><code>self.weight</code> is a probability. For example, <code>0.5</code> means my batch has half the samples from inside and half the samples from outside. I have two <code>randint</code> calls and a <code>permute</code> which makes yielding a batch slow. I am working with batch sizes of 128/256 per GPU. I think I can create a flatten array of length 82 million where every slice of 128/256 samples follow a distribution where half of the samples are from in and half are from out. But every attempt I try to this is prohibitively slow. Any idea how to speed up the dataloader?</p>
Mismatched number of neurons in classification layer and number of classes

username: idgx
cooked: <pre><code>class NeuralNetwork(nn.Module):
    def __init__(self):
    super().__init__() 
    self.conv1 = nn.Conv2d(1, 20, stride = 1, kernel_size = 5) # Makes 20 maps of 24x24
    self.pool = nn.MaxPool2d(2, 2)                             # Makes 20 maps of 12x12
    self.fc1 = nn.Linear(20 * 12 * 12, 100) #  &lt;---                
    
def forward(self, x):
    x = self.pool(torch.sigmoid(self.conv1(x)))
    x = torch.flatten(x, 1) # flatten all dimensions except batch
    x = self.fc1(x)        
    return x
</code></pre>
<p>I’m trying to train this network with MNSIT dataset and I do not get any tensor size mismatch error!<br>
In the last layer, I have 100 neurons instead of having 10! In my case, accuracy was 1% worse with 100 output neurons instead of 10, I also see that <a href="https://discuss.pytorch.org/t/pre-trained-model-and-finetuning/36418/10">this</a> issue was listed. Anyway, my concern is that why was the size mismatch not flagged ? How is autograd dealing with this situation ? Any insights that will shed light on this will be appreciated.</p>

username: ptrblck
cooked: <p>Your model is allowed to output more logits than you are using. I.e. in your example logit 10-99 will contain “random” data for the unknown classes with indices 10-99. The loss calculation doesn’t fail since targets are valid as long as they are in <code>[0, nb_classes-1]</code>. which is the case if you artificially inflate <code>nb_classes</code>.</p>

username: akt42
cooked: <p>Could you show how the loss function was called and the dimensions of the predictions and labels tensors?</p>

username: idgx
cooked: <p>I am using <code>loss_fn = nn.CrossEntropyLoss()</code> . Therotically, loss is given by -ln(Softmax(activation of neuron corresponding to the label)).</p>
<p>Dataset was ontained like below:</p>
<pre><code>training_data = datasets.MNIST(root = "data",train = True,download = True,transform = ToTensor()) 
test_data = datasets.MNIST(root = "data",train = False,download = True,transform = ToTensor())
</code></pre>
PyTorch equivalent of "Gaussian Mixture Models" implemented in Scikit-Learn"

username: oat
cooked: <p>I want to use <strong>Gaussian Mixture Models initiated with K-Means</strong> to do <strong>cluster analysis</strong> for a data set with 6 features, i.e. <strong>unsupervised learning to detect potential classes, or groups, in the data set</strong>.</p>
<p>I know how to do this through <a href="https://scikit-learn.org/stable/modules/generated/sklearn.mixture.GaussianMixture.html?highlight=gaussianmixture" rel="noopener nofollow ugc">Gaussian Mixture Models</a> in Scikit-Learn, as shown below:</p>
<pre><code class="lang-auto"># init GMM with K-Means
gm_kmeans = GaussianMixture(
    n_components = 30, 
    max_iter= 1000, 
    tol = 1e-4, 
    init_params= 'kmeans', 
)

# predicated class
y = gm_kmeans.fit_predict(data.iloc[:, 2:8])    

# predicted probabilities belonging to each of the classes
y_proba = gm_kmeans.predict_proba(data.iloc[:, 2:8])   
</code></pre>
<p>However, this algorithm usually takes a long time to calculate via CPU, and Scikit-Learn is not designed to utilize GPU for parallel processing in this regard.</p>
<p>So, I’d like to ask if there is a PyTorch equivalent to this algorithm. Or, how to implement Gaussian Mixture Models init with K-Means for unsupervised classification that can utilize GPU.</p>
<p>Homework I did:</p>
<ol>
<li><a href="https://discuss.pytorch.org/t/parallel-implementation-of-gaussian-mixture-models-for-working-with-multiple-gpus/49074">Parallel Implementation of Gaussian mixture models for working with multiple GPU’s</a></li>
<li><a href="https://stackoverflow.com/questions/65755730/estimating-mixture-of-gaussian-models-in-pytorch" rel="noopener nofollow ugc">Estimating mixture of Gaussian models in Pytorch</a></li>
</ol>

username: oat
cooked: <p>Any advice is greatly appreciated.</p>

username: pumplerod
cooked: <p>Any luck finding this <a class="mention" href="/u/oat">@oat</a> ?</p>
<p>I came across <a href="https://github.com/ldeecke/gmm-torch" rel="noopener nofollow ugc">gmm-torch</a> however it seems to not take advantage of the autograd system of pytorch.  I’d also like to find a demonstration for how to implement this in pytorch using a traditional training of parameters with optimizers.</p>
Load large multivariate time-series dataset with TorchData

username: nleh
cooked: <p>Hello,</p>
<p>I have a large multivariate time-series dataset in a .parquet file with which I want to do forecasting with the <a href="https://github.com/jdb78/pytorch-forecasting" rel="noopener nofollow ugc">pytorch-forecasting</a> library. The problem I am having is that the library assumes that your dataset fits into memory in a pandas dataframe to conduct dataloading. The dataset consists of a <code>group_idx</code>, <code>time_idx</code> and several covariates. The <code>group_idx</code> identifies a respective individual timeseries and <code>time_idx</code> is a sequential index of time steps.</p>
<p>To tackle the memory problem, I have looked at TorchData and DataPipes, but there is only a simple example about univariate timeseries data from a small csv file. I am hoping to achieve the following:</p>
<ol>
<li>load data without running into memory issues</li>
<li>perform some normalization procedures of continuous and categorical variables, like target normalization and encoding categorical variables numerically</li>
<li>from a larger individual timeseries sequence extract a subsequence that respects the encoding and decoding length</li>
<li>possibly handle missing time steps within subsequence with a simple imputation scheme</li>
<li>create training and validation split that respects the time horizon and ensures that there is no leak of information in the temporal dimension</li>
</ol>
<p>Essentially, attempt to build a timeseries data pipeline that has similar capabilities like the pytorch-forecasting Dataset class but instead enable large scale datasets that do not fit into memory.</p>
<p>Looking at TorchData, a starting point for me is the <code>ParquetDataFrameLoader</code> but I am not sure how to use it in conjunction with the other necessary steps, for example only querying an individual timeseries to form a sample that can be loaded in batches.</p>
<p>I have thought about a more traditional <code>Dataset</code> approach, where to saved individual time series separately and build a file index to use the “standard” pytorch dataloading way, but there are more than 200k groups in my dataset so saving them all individually is also inconvenient. I am therefore wondering about suggestions about how to do this with TorchData and what the individual pieces should be or whether it is even possible currently. And if not if you have suggestions of other ways or libraries that could tackle this issue. Thank you in advance.</p>

username: alaameloh
cooked: <p>Hey there <a class="mention" href="/u/nleh">@nleh</a></p>
<p>Hope you managed to fix your problem<br>
I’m currently facing the same issue, and was wondering how did you manage eventually ?</p>
Current state of the torchvision.transformation

username: aguennecjacq
cooked: <p>Hi everyone,<br>
I’ve just come back to pytorch after 2 years of doing other stuff.<br>
Does anyone know about the current state/future changes of the  torchvision.transformation ?<br>
I read in github merge request#6233  that the tranforms API was currently under review but I haven’t found any recent news about it. Furthermore, it seems that in v2, there is now a transformation class !<br>
Thanks in advance,<br>
Antoine</p>
Invalid CUDA 'device=0' requested. Use 'device=cpu' or pass valid CUDA device(s) if available, i.e. 'device=0' or 'device=0,1,2,3' for Multi-GPU

username: Andrea_Basile
cooked: <p>Hi!<br>
I am trying to use YOLOv8 with my Jetson Orin. I am trying to run python code but this error occurs:<br>
WARNING <img src="https://discuss.pytorch.org/images/emoji/apple/warning.png?v=12" title=":warning:" class="emoji" alt=":warning:" loading="lazy" width="20" height="20"> TensorRT requires GPU export, automatically assigning device=0<br>
Ultralytics YOLOv8.0.199 <img src="https://discuss.pytorch.org/images/emoji/apple/rocket.png?v=12" title=":rocket:" class="emoji" alt=":rocket:" loading="lazy" width="20" height="20"> Python-3.8.10 torch-2.1.0<br>
Traceback (most recent call last):<br>
File “export.py”, line 29, in <br>
main()<br>
File “export.py”, line 25, in main<br>
model.export(format = ‘engine’)<br>
File “/media/jetson_orin/orin_ssd/yolov8env/lib/python3.8/site-packages/ultralytics/engine/model.py”, line 313, in export<br>
return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model)<br>
File “/media/jetson_orin/orin_ssd/yolov8env/lib/python3.8/site-packages/torch/utils/_contextlib.py”, line 115, in decorate_context<br>
return func(*args, **kwargs)<br>
File “/media/jetson_orin/orin_ssd/yolov8env/lib/python3.8/site-packages/ultralytics/engine/exporter.py”, line 172, in <strong>call</strong><br>
self.device = select_device(‘cpu’ if self.args.device is None else self.args.device)<br>
File “/media/jetson_orin/orin_ssd/yolov8env/lib/python3.8/site-packages/ultralytics/utils/torch_utils.py”, line 119, in select_device<br>
raise ValueError(f"Invalid CUDA ‘device={device}’ requested."<br>
ValueError: Invalid CUDA ‘device=0’ requested. Use ‘device=cpu’ or pass valid CUDA device(s) if available, i.e. ‘device=0’ or ‘device=0,1,2,3’ for Multi-GPU.</p>
<p>torch.cuda.is_available(): False<br>
torch.cuda.device_count(): 0<br>
os.environ[‘CUDA_VISIBLE_DEVICES’]: *<br>
See <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a> for up-to-date torch install instructions if no CUDA devices are seen by torch.</p>
<p>I tried all suggestions that I read on this forum but nothing changed.<br>
Thanks for your reply and help!</p>

username: ptrblck
cooked: <p>Your PyTorch installation doesn’t support CUDA so install the binaries from e.g. <a href="https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048">here</a> as described in similar issues regarding Jetson devices in this forum.</p>