akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
System shuts down automatically when training model

BeeManJunior: <p>I am using Ubuntu 18.04. I got NVIDIA GTX 1060 with nvidia drivers v415. Every time I train the model on GPU while on battery power, system automatically shuts down. But, when I am connected to power, it doesnt happen. I am not sure as to what is causing this problem. Has anyone faced such a problem ?<br>
Any help would be appreciated.</p>

ptrblck: <p>Was the system working before?<br>
Does your laptop restart, crash or goes to sleep mode?<br>
Could you check the battery status in the settings?<br>
Did you upgrade your laptop with the GTX 1060 or was it already built in?</p>

BeeManJunior: <p>I use Dell G7 with a built in NVIDIA GTX 1060. Couple of times the “Power Off” dialog box popped up and it immediately shut down, and sometimes it shuts down abruptly(may be crashing) throwing a black screen. But, when I’m plugged into a power source, it runs perfect.</p>

ptrblck: <p>Your PSU might be a problem, if it was working before.<br>
Also, have you checked if your laptop uses the GPU at all when in battery mode?<br>
I’m not familiar with this specific machine, but I would guess some laptops might switch to the on-board graphic processor when not plugged in to save energy.</p>

scarecrow21: <p>The same is happening with me. Lenovo Legion y-540, Ubuntu 18, RTX-2060(6gb). My laptop automatically shuts down after a few epochs of training on GPU. I was training a multi-class text classification model using RNN+CNN. I think it’s a memory issue. But in my case, I have set the GPU only for CUDA. My laptop graphic runs on Intel Integrated HD card.</p>
<p>I don’t really know why this happens. And it only happens for multi-class text classification and not binary text classification (possibly a memory issue).</p>
<p>Any leads on this?</p>

raj-rishav: <p>Any update on this, as I am having the same laptop. I was able to train models on GPU while using Ubuntu 18, but since I installed Ubuntu 20, its shutting down as I start to train a model.</p>
EigenDecomposition of a matrix

AISSA_OUBOUKIOUD: <p>Hi guys, I am using pytorch to calculate the eigendecomposition of a matrix</p>
<pre><code class="lang-python">&gt;&gt;&gt; import torch
&gt;&gt;&gt; a = torch.tensor([[1, 2, -2], [2, 1, -2], [2, 2, -3]], dtype=torch.float32)
&gt;&gt;&gt; torch.linalg.eig(a)
torch.return_types.linalg_eig(
eigenvalues=tensor([ 1.0000+0.j, -1.0000+0.j, -1.0000+0.j]),
eigenvectors=tensor([[-5.7735e-01+0.j, -3.4413e-08+0.j,  3.4246e-01+0.j],
        [-5.7735e-01+0.j,  7.0711e-01+0.j,  4.7068e-01+0.j],
        [-5.7735e-01+0.j,  7.0711e-01+0.j,  8.1313e-01+0.j]]))
</code></pre>
<p>Is there anyway to get the eigenvalues and the eigenvectors as whole number ??</p>

AlphaBetaGamma96: <p>if you mean as reals (as <code>torch.linalg.eig</code> returns them as dtype <code>torch.complex64</code>), you can just do</p>
<pre><code class="lang-auto">eigenvalues = torch.real(eigenvalues)
eigenvectors = torch.real(eigenvectors)
</code></pre>
<p>and that’ll convert them to reals</p>
Inner product 2-D weight matrix with 3-D torch of spectrograms

Noy_Uzrad: <p>Hi.</p>
<p>I have the following setup:<br>
[49, 49] matrix, where each row is a probabilities vector (obtained from softmax over logits). overall it has 49 probability vectors, each with 49 examples.</p>
<p>[49, x, y] matrix, containig 49 spectrograms of size [x,y] each.</p>
<p>I try to obtain a 49 different weighted spectrograms, from each of the 49 probability vectors and 49 spectrograms.</p>
<p>Output size shall be [49, x, y].</p>
<p>I tried my best to search the net, and tried many configurtations of torch matmul, bmm, etc…</p>

InnovArul: <aside class="quote no-group" data-username="Noy_Uzrad" data-post="1" data-topic="138232">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/noy_uzrad/48/44471_2.png" class="avatar"> Noy_Uzrad:</div>
<blockquote>
<p>Output size shall be [49, x, y].</p>
</blockquote>
</aside>
<p>If I understand correctly, since you want differently weighted spectrograms with each of the 49 probability vectors and 49 spectrograms, the output size should be [49, 49, x, y]. No?</p>
<p>i.e., In the spectrogram size <code>[49, x, y]</code>, batch size = 49. The same batch size as in the probability matrix of <code>[49, 49]</code>. For a particular spectrogram, you have 49 probability values corresponding to different classes.</p>

Noy_Uzrad: <p>Ok I managed to figure this out <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
with the following:<br>
torch.einsum(“ik,klm-&gt;ilm”, probabilities, spectrograms)</p>
<p><a class="mention" href="/u/innovarul">@InnovArul</a> you undertood me wrong. Each of the rows in the probabilities matrix is a probability vector with 49 values. Now I multiply in an inner-product manner with the 49 spectrograms to get a weigted spectrogram of size [x, y]. Since the probability matrix has 49 rows I get an overall 49 weighted spectrograms so out size is [49, x, y]</p>
<p>I just realized einsum and this is a SUPER powerful tool! I highly recommend to any deep learning practitioner to use it</p>
Reversed index mapping in c++/cuda extension?

YutongZheng: <p>I’m trying to implement a 3D resampling kernel with a sparse feature map input. The basic idea is shown in my PyTorch script:</p>
<pre><code class="lang-auto">def forward(feat, loc, new_loc, K): # forward function of the 3D resampling layer.
    '''feat:     float tensor at size (N,C). N is the number of non-empty locations in the 3D space. 
       loc:      int64 tensor at size (N,), which keeps track of the absolute location in the 3D space. It is int64 encoded because we use fixed grid sampling for the input feature map. 
       new_loc:  float tensor at size (M,), which indicates the new (continuous) locations to be interpolated. 
       kernel:   float tensor at size (K,), the 3D resampling kernel.
    '''
    ...
    # for each new location, find the absolute location of the input feature within the kernel. 
    loc_in_kernel = find_neighboring(new_loc, kernel.shape[0]) # int64 size of (M,K)

    # find the reversed mapping, such that loc[reversed_idx] = loc_in_kernel.
    reversed_idx = find_a_in_b(a=loc_in_kernel, b=loc) # int64 size of (M,K)

    # apply the parallelism resample kernel in 3D space. (No memory issue here.)
    new_feat = resample3d(feat, kernel, reversed_idx) # size of (M,C)
    return new_feat
</code></pre>
<p>In practice, K can be really large, e.g. 10x10x10=1000, which makes the (M, K) tensors easily out of memory. One solution is to implement the <code>find_neighboring()</code> and <code>find_a_in_b()</code> functions in Cuda and incorporate with <code>resample3d()</code> function. The parallelism will be applied to dimension size M.</p>
<p>However, in c++/cuda, I cannot figure out a way to do the reversed index mapping as <code>find_a_in_b()</code>. Also, I’m not sure if using for-loops to find the reversed index, in this case, will be super slow (Since N can also be at the size of millions)? Can anyone help?</p>

A_A: <p>Not a complete answer but you should probably take a look into <a href="https://docs.nvidia.com/cuda/thrust/index.html" rel="noopener nofollow ugc">thrust</a> library, it offers lots of building blocks for such operations on GPU.</p>
Mapping the values in a tensor to another unique tensor

YutongZheng: <p>Consider there are two int64 torch::Tensor x and y at the same length in the c++ implementation, both with unique values. Could you kindly help me to create a mapping M() between the two, such that M(a[i]) = b[i]? Thanks!</p>

A_A: <p>Not aware of a solution via PyTorch but you can possibly make use of <a href="https://docs.nvidia.com/cuda/thrust/index.html" rel="noopener nofollow ugc">thrust</a> library.</p>
<p>A crude logic would be to first both arrays by <a href="https://thrust.github.io/doc/group__sorting_gabe038d6107f7c824cf74120500ef45ea.html#gabe038d6107f7c824cf74120500ef45ea" rel="noopener nofollow ugc">sort_by_key</a> operation where keys are the x, y and values are the indices. Afterwards you have a mapping from sorted indices in x to sorted indices in y which can be inverted.</p>
Torchaudio.transforms doesn't have log scale frequency spectrogram

Eri24816: <p>Hi,<br>
I’ve found <strong>Spectrogram</strong> and <strong>MelSpectrogram</strong> in torchaudio.transforms, but there seems to be no implementation of <strong>log scale frequency spectrogram</strong>.<br>
Why is that?</p>

Noy_Uzrad: <p>Hi. Look at AmplitudeToDb <a href="https://pytorch.org/audio/stable/transforms.html#amplitudetodb" rel="noopener nofollow ugc">https://pytorch.org/audio/stable/transforms.html#amplitudetodb</a><a href="https://pytorch.org/audio/stable/transforms.html#amplitudetodb" class="inline-onebox" rel="noopener nofollow ugc">torchaudio.transforms — Torchaudio 0.10.0 documentation</a></p>
Patchify images dynamically in pytorch dataset

Oliver: <p>I’m using the <code>patchify</code> pypi module to first <code>patchify</code> images and save them as a new dataset and then load them back as individual images in a pytorch dataset.<br>
It’s just:</p>
<pre><code>`patches_img = patchify(image, (patch_size, patch_size, 3), step=patch_size)`
</code></pre>
<p>for all images and then I’ll load the individual patches in just like any images.</p>
<p>What I’d like to do is something like this</p>
<pre><code class="lang-auto">    class Dataset(object):
        #images are the full pre patchified images
        def __init__(self, images: List[str]):
            self.images = images 
            self.transformer = Transform()
            
        
        def __getitem__(self, idx: int):
            image = self.images[idx]
            #IMPORTANT switch to color later
            patches = patchify(image, (patch_size, patch_size, 3), step=patch_size)
    
            image_patches_tensor = self.transformer(patches)
            return image_patches tensor
        
        def __len__(self):
            return len(self.files)
</code></pre>
<p>how would I do that?</p>
<p>Thanks in advance!</p>

arman-yekkehkhani: <p>Are all images the same size? If the answer is yes, we can assume that each image is patched into <code>P</code> patches and there are <code>N</code> original images. Then  you can modify the dataset to be sth like this:</p>
<pre><code class="lang-auto">  class Dataset(object):
        #images are the full pre patchified images
        def __init__(self, images: List[str], P):
            self.images = images 
            self.transformer = Transform()
            self.num_patches = P
            
        
        def __getitem__(self, idx: int):
            img_idx = idx // self.num_patches
            patch_idx = idx % self.num_patches
            image = self.images[img_idx]
            #IMPORTANT switch to color later
            patches = patchify(image, (patch_size, patch_size, 3), step=patch_size)
    
            image_patches_tensor = self.transformer(patches[patch_idx])
            return image_patches tensor
</code></pre>

Oliver: <p>Hi arman,</p>
<p>thanks, this is something I did already, I was to use pytorch’s native unfold and have it done on the fly as transformatinos.<br>
They are not the same size, but above works regardless if use // integer division by patch_size.</p>
<p>thanks for the reply, apologies for overlooking it for so long.</p>
ModuleList typing error: not an iterable

Pascal1: <p>If I use a ModuleList:</p>
<pre><code class="lang-auto">import torch.nn as nn

class Model(nn.Module):

    def __init__(self):
        super().__init__()
        self.module_list = nn.ModuleList(
            [nn.Linear(8, 8), nn.Linear(8, 8), nn.Linear(8, 8), nn.Linear(8, 8), nn.Linear(8, 1)]
        )

    def forward(self, batch):
        for i in self.module_list[1:4]:
            pass
</code></pre>
<p>I get a typing error: Expected type ‘Iterable’ (matched generic type ‘Iterable[_T1]’), got ‘Module’ instead.<br>
How do I fix this?<br>
(I would like to avoid creating multiple nn.ModuleLists)</p>
<p>Thanks.</p>
<p>Best</p>
<p>Pascal</p>

ptrblck: <p>Which PyTorch version are you using as your code works in my setup (<code>1.11.0.dev20211101+cu113</code>):</p>
<pre><code class="lang-python">class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.module_list = nn.ModuleList(
            [nn.Linear(8, 8), nn.Linear(8, 8), nn.Linear(8, 8), nn.Linear(8, 8), nn.Linear(8, 1)]
        )

    def forward(self, batch):
        for i in self.module_list[1:4]:
            print(i)
        return batch

model = Model()
out = model(torch.randn(1, 1))
print(out.shape)
&gt; torch.Size([1, 1])
</code></pre>

Pascal1: <p>Thanks for your quick reply.<br>
I am using ‘1.10.0+cpu’.<br>
Code also runs for me, but I get a typing error. i.e. mypy would complain.</p>

ptrblck: <p>Ah, OK, sorry as I misunderstood the error.<br>
Unfortunately, I don’t know how to fix the mypy typing error/warning.</p>

Pascal1: <p>Took a look into the code, I guess this is a bug in:<br>
/nn/modules/container.py</p>
<pre><code class="lang-auto">def __getitem__(self, idx: int) -&gt; Module:
        if isinstance(idx, slice):
            return self.__class__(list(self._modules.values())[idx])
        else:
            return self._modules[self._get_abs_string_index(idx)]
</code></pre>
<p>The return type is actually not Module but Union[Module, ‘ModuleList’].<br>
Should I open a PR for this?</p>

ptrblck: <p>Yes, I’m sure your fix is more than welcome! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Help with specifying parameters

ofir_ber: <p>Hello,</p>
<p>I’m pretty new to deep learning, I would be happy to receive any kind of help.<br>
I have an exercise that I need to train a Convolutional Neural Network (CNN) on the CIFAR10 dataset. After building 31 layers I have these layers:</p>
<p>self.conv10 = nn.Conv2d(502, 502, 3, padding=1)<br>
self.bn10 = nn.BatchNorm2d(502)<br>
self.relu10 = nn.ReLU()<br>
self.maxpool5 = nn.MaxPool2d(2, stride = 2)<br>
self.avgpool1 = nn.AdaptiveAvgPool2d(???)<br>
self.dropout = nn.Dropout(0.48254838509851644)<br>
self.fc1 = nn.Linear(??? ,2039)<br>
self.relu11 = nn.ReLU()<br>
self.fc2 = nn.Linear(2039,???)</p>
<p>there are three parameters that I need to compute (all the ???) - one in the AdaptiveAvgPool2d, one in the input of the first Linear, and one in the output of the second Linear. How exactly do I compute these parameters?</p>
<p>Thanks</p>
Depthwise Convolution Memory Leakage

LyricZhao: <p>Hi, I have a memory leakage problem with the code below:</p>
<pre><code class="lang-python">import torch
import torch.nn as nn


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.c, self.k, self.p, self.x = 256, 3, 1, 2304
        self.n, self.h, self.w = 0, 0, 0
        self.unfold = nn.Unfold(self.k, padding=self.p)
        self.conv = nn.Conv2d(self.c * self.k * self.k, self.x, 1, padding=0, groups=self.c * self.k * self.k)

    def forward(self, x: torch.Tensor):
        self.n, self.h, self.w = x.size(0), x.size(2), x.size(3)
        c1 = x.view(self.n, self.c, 1, self.h, self.w)

        c2 = self.unfold(x)
        c2 = c2.view(self.n, self.c, self.k * self.k, self.h, self.w)

        out = c1 + c2
        out = out.view(self.n, self.c * self.k * self.k, self.h, self.w)
        return self.conv(out)


if __name__ == '__main__':
    try:
        net = Net().cuda()
        x = torch.randn((1024, 256, 32, 32)).cuda()
        out = net(x)
        print(out)
    except Exception as ex:
        print(f' &gt; CUDA memory: {torch.cuda.memory_allocated() / 1024 ** 3} GiB')
        del x, net
        print(ex)
    print(f' &gt; CUDA memory: {torch.cuda.memory_allocated() / 1024 ** 3} GiB')
</code></pre>
<p>Here I define a network with depth-wise convolution as the last layer. I’m using 24GB RTX 3090, so when it runs into <code>self.conv(out)</code>, it will surely be out of CUDA memory due to the large tensors. Then we can catch the exception and delete the references of <code>x</code> and <code>net</code>, the expected CUDA memory after <code>del x, net</code> should be zero but I got 9 GB instead, which means that there’s a memory leakage with a failed-to-allocate-output depth-wise convolution. If I change the group number of <code>self.conv</code> into something else, the CUDA memory will be correctly zero.</p>

ptrblck: <p>I get the expected result:</p>
<pre><code class="lang-python"> &gt; CUDA memory: 19.000017166137695 GiB
CUDA out of memory. Tried to allocate 9.00 GiB (GPU 0; 23.69 GiB total capacity; 19.00 GiB already allocated; 1.48 GiB free; 19.00 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
 &gt; CUDA memory: 0.0 GiB
</code></pre>
<p>but think that these “hacks” might easily fail (as seen in your setup) and would thus recommend to set a proper batch size before starting the training.<br>
In any case, you could check if deleting <code>out</code> (if it’s even created) might solve your issue.</p>
State_dict manipulation for model split

tungnt: <p>Hi. I want to split a pretrained model, Torchvision’s faster RCNN to be particular, into two parts. However, the model is fully sequential.</p>
<pre><code class="lang-auto">FasterRCNN(
  (transform): GeneralizedRCNNTransform(
      Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
      Resize(min_size=(800,), max_size=1333, mode='bilinear')
  )
  (backbone): BackboneWithFPN(
    (body): IntermediateLayerGetter(
      (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
      (bn1): FrozenBatchNorm2d(64, eps=0.0)
      (relu): ReLU(inplace=True)
      (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
     # This is where I wanna split the model.
</code></pre>
<p>Currently, my very hacky solution is to go layer-by-layer and get the sub modules (children) by hand.</p>
<pre><code class="lang-auto">transform = model.get_submodule(target = 'transform')
backbone = model.get_submodule(target = 'backbone')
backbone_body = backbone.get_submodule(target = 'body')
body_layers = list(backbone_body.children())[0:4]
</code></pre>
<pre><code class="lang-auto">new_model = torch.nn.Module()
new_model.add_module('transform', transform)
new_backbone = torch.nn.Module()
new_backbone_body = torch.nn.ModuleDict()
layer_names = ['conv1', 'bn1', 'relu', 'maxpool']
for l_name, layer in zip(layer_names, body_layers):
    new_backbone_body.add_module(l_name, layer)

new_backbone.add_module('body', new_backbone_body)
new_model.add_module('backbone', new_backbone)
</code></pre>
<p>which gave me this new model:</p>
<pre><code class="lang-auto">Module(
  (transform): GeneralizedRCNNTransform(
      Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
      Resize(min_size=(800,), max_size=1333, mode='bilinear')
  )
  (backbone): Module(
    (body): ModuleDict(
      (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
      (bn1): FrozenBatchNorm2d(64, eps=0.0)
      (relu): ReLU(inplace=True)
      (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
    )
  )
)
</code></pre>
<p>When I loaded the pretrained weights into the new model with <code>strict = True</code>, I didnt find any of the layers that I had just manually added in the error message. Do you think that this was successful? Or do you think there could be any potential issues during deployment?</p>
<p>Any thought would be much appreciated.<br>
Thanks,</p>
Memory_allocated

DDavid: <pre><code class="lang-auto">&gt;&gt;&gt; out = torch.randn(100,100)
&gt;&gt;&gt; out.cuda()
&gt;&gt;&gt; torch.cuda.memory_allocated(0)
40448
&gt;&gt;&gt; out1 = torch.randn(100,100)
&gt;&gt;&gt; torch.cuda.memory_allocated(0)
0
</code></pre>
<p>Why is this happening? I expected the output of to<code>torch.cuda.memory_allocated(0)</code> be 40448 still, because <code>out1</code> is totally different to <code>out</code> .<br>
Note that,</p>
<pre><code class="lang-auto">&gt;&gt;&gt; id(out)
139989884735632
&gt;&gt;&gt; id(out1)
139989884693456
</code></pre>
<p>Note2</p>
<pre><code class="lang-auto">&gt;&gt;&gt; out = torch.randn(100,100)
&gt;&gt;&gt; out.cuda()
&gt;&gt;&gt; out1 = torch.randn(90,90)
&gt;&gt;&gt; torch.cuda.memory_allocated(0)
40448
</code></pre>

ptrblck: <p>I cannot reproduce the issue and get:</p>
<pre><code class="lang-python">out = torch.randn(100,100)
out.cuda()
print(torch.cuda.memory_allocated(0))
&gt; 0

out1 = torch.randn(100,100)
print(torch.cuda.memory_allocated(0))
&gt; 0
</code></pre>
<p>Are you using a REPL environment where the output of the last operation might still be stored in the <code>_</code> object?</p>

DDavid: <p>Yes I was using the REPL env. Is there a docs/blog posts explaining what you just said here : <code>the output of the last operation be stored in the _ object</code>?</p>
<p>Thanks in advance.</p>

ptrblck: <p>The underscore (<code>_</code>) is a special variable which will get the output of the last operation as seen here:</p>
<pre><code class="lang-python">a = 5

a
Out[2]: 5

6
Out[3]: 6

_
Out[4]: 6

_ + _
Out[5]: 12
</code></pre>
<p>If you are stepping through the code in a REPL env I would guess that in particular this variable can hold a reference to the CUDATensor and could thus report the GPU usage. On the other hand, if you execute the entire script, the memory usage is shown as <code>0.</code> which is expected since you didn’t assign the CUDATensor to any variable so it will be discarded.</p>
Wrong gradients in quantization-aware training

sherylwang: <p>Hi, I use _fake_quantize_learnable_per_tensor_affine api as a component of my quantization layer. I found in some cases, this api will make the gradients of scale, zero_point and input wrong. The gradients generated by backward is different from the ones calculated by hand, even generated nan error. After I replaced  _fake_quantize_learnable_per_tensor_affine with my own written code, the nan disappeared. My pytorch version 1.8.2.</p>

sherylwang: <p>Hi, is there any help?</p>

supriyar: <p>Hi <a class="mention" href="/u/sherylwang">@sherylwang</a> , do you have an example that could reproduce the behavior? Typically a small representative model would help us understand and debug the problem better.</p>
Very poor Batch Sizes on RTX 3090, good Batch Sizes on RTX Titan (Both 24gb cards)

seanfarr788: <p>I have a local machine which has an RTX 3090, and also have access to a GPU server with a RTX Titan, both have 24gb VRAM, both have identical amounts of RAM/same CPU’s etc.</p>
<p>I am running the exact same code, installed same versions of PyTorch (Nightly) and all my packages, and yet the Titan is able to have batch sizes of 32 whilst on the 3090 it is unable to go as high as 8 where I get a CUDA out of memory error.</p>
<p>Any suggestions greatly appreciated (first time poster so apologies if I am doing something wrong)<br>
Many Thanks</p>

my3bikaht: <p>That shouldn’t be an issue. Have you checked what amount of memory python consumes during batch 8 run? (via nvidia-smi for example)</p>

seanfarr788: <p>Hi <a class="mention" href="/u/my3bikaht">@my3bikaht</a>,</p>
<p>Many thanks for replying, I tried installing the latest Nvidia Drivers (495.44) on the latest feature branch which seems to be able to increase my batch size to 16 which is at least an improvement and somewhat closer to the cluster.</p>
<p>A batch size of 16 = 16.8gb, batch size of 32 gets an out of memory error locally, but 23888mb on the cluster. I suspect now that as the cluster does not have any display outputs/other processes etc. it has more available VRAM than my local machine and hence the reduced batch size.</p>

eqy: <p>It might be useful to show the output of <code>nvidia-smi</code> (which would indicate memory usage by the desktop environment and other potential processes).</p>
nn.Mish second derivative

ipostr08: <p>When I try to get a Hessian of a net using nn.Mish on a GPU I get Nans. I see that exp() is used in the C++ code, which could be the reason. Is being able to get the second derivative of various internally implemented functions something expected or not?</p>

AlphaBetaGamma96: <p>You can manually implement a custom function within the python API via <code>torch.custom.autograd</code> (with a second derivative as well)<br>
<a href="https://pytorch.org/docs/stable/autograd.html?highlight=unction#torch.autograd.Function" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/autograd.html?highlight=unction#torch.autograd.Function</a></p>

ipostr08: <p>Yes, that’s what I did. But I’m wondering about the general philosophy regarding second derivatives of various internal functions since some of them are faster and more thoroughly examined than custom Python implementations.</p>
2D differentiable rendering?

ZimoNitrome: <p>Are there any libraries available for 2D differentiable rendering of shapes (i.e. circles, lines, polygons) or similar structures?</p>

Flo_no: <p>Yes <a href="https://github.com/BachiLi/diffvg" class="inline-onebox" rel="noopener nofollow ugc">GitHub - BachiLi/diffvg: Differentiable Vector Graphics Rasterization</a></p>
Backward pass through ONNX object

Amit_Gupta: <p>I would like to make an onnx model differentiable. As I understand exporting to ONNX does not export the AutoGrad graph, is there anyway to reconstruct if after loading?</p>
<p>I am aware of torch-ort but to me it looks like it only works with nn.Module objects, i.e. original python pytorch models ? (see example <a href="https://github.com/microsoft/onnxruntime-training-examples" rel="noopener nofollow ugc">here</a>, <a href="https://techcommunity.microsoft.com/t5/azure-ai-blog/accelerate-pytorch-transformer-model-training-with-onnx-runtime/ba-p/2540471" rel="noopener nofollow ugc">here</a>  and <a href="https://cloudblogs.microsoft.com/opensource/2021/07/13/accelerate-pytorch-training-with-torch-ort/" rel="noopener nofollow ugc">here</a>)</p>
<p>Can I in any way load an ONNX exported model and get pytorch or onnx-runtime to reconstruct the backward graph?<br>
<small><br>
Alternatively, can I get onnx to export backward graph of a PyTorch nn.Module model? So that I can run it with onnx runtime?</small></p><small>
</small><p><small>Background: I want to work with physics based models, where I could easily write forward “energy” function, and can use their gradient (“forces”) in my simulations. At present we need either numerical diff, or analytic expressions to be derived before hand.<br>
</small></p>
Unicode Normalization with Torchscript

anjali-chadha: <h2>
<a name="questions-and-help-1" class="anchor" href="#questions-and-help-1"></a><img src="https://discuss.pytorch.org/images/emoji/apple/question.png?v=10" title=":question:" class="emoji" alt=":question:"> Questions and Help</h2>
<p>I am writing a AlbertTokenizer based preprocessor which can be serialized using Torchscript.</p>
<p>One of the preprocessor steps is doing unicode normalization (ref implementation in HFs’s <a href="https://github.com/huggingface/transformers/blob/master/src/transformers/models/albert/tokenization_albert.py#L210" rel="noopener nofollow ugc">AlbertTokenizer</a>)</p>
<pre><code class="lang-auto">outputs = unicodedata.normalize("NFKD", outputs)
</code></pre>
<p>However, when I try to serialize the preprocesor with this code, I get following error</p>
<pre><code class="lang-auto">Python builtin &lt;built-in function normalize&gt; is currently not supported in Torchscript:
  File "&lt;ipython-input-9-dcefac57d541&gt;", line 53
        outputs = text
        if not self.keep_accents:
            outputs = unicodedata.normalize("NFKD", outputs)
                      ~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
    #             outputs = "".join([c for c in outputs if not unicodedata.combining(c)])
        return outputs

</code></pre>
<p>What is the recommended way to perform Unicode Normalization while making sure the code is serializable?</p>

Michael_Suo: <p>The best way to do this is to use the custom operator API to bind in a unicode normalization op.</p>

anjali-chadha: <aside class="quote no-group" data-username="Michael_Suo" data-post="2" data-topic="138160">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/7993a0/48.png" class="avatar"> Michael_Suo:</div>
<blockquote>
<p>custom operator API</p>
</blockquote>
</aside>
<p>Thank you <a class="mention" href="/u/michael_suo">@Michael_Suo</a> for your response. Is there any reference example that you can share for writing such custom operator?</p>

Michael_Suo: <p>This tutorial is the best place to start: <a href="https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html" rel="noopener nofollow ugc">https://pytorch.org/tutorials/advanced/torch_script_custom_ops.html</a>. Of course it is not about unicode normalization; you may be have to find a cpp library that implements it (similar to what is done with opencv in the tutorial)</p>
Load matlab files in pytorch

toneldnp: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/1/61822a2402050af5b73b36d45d17311531216cfd.png" data-download-href="https://discuss.pytorch.org/uploads/default/61822a2402050af5b73b36d45d17311531216cfd" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61822a2402050af5b73b36d45d17311531216cfd_2_97x500.png" alt="image" data-base62-sha1="dUBbQjqPTXkBHG5mYxN4rrfMU1f" width="97" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61822a2402050af5b73b36d45d17311531216cfd_2_97x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61822a2402050af5b73b36d45d17311531216cfd_2_145x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/1/61822a2402050af5b73b36d45d17311531216cfd_2_194x1000.png 2x" data-dominant-color="39393A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">238×1222 30.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I have these .mat files and I want to load them as trainsets.</p>

eqy: <p>I believe this method might work here:</p><aside class="quote" data-post="2" data-topic="38094">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-convert-mat-file-struct-into-pytorch-tensors/38094/2">How to convert .mat file struct into pytorch Tensors</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    I guess you’ve opened the file using scipy.io? 
If so, you should get a dict with all the members and can just load these arrays as numpy arrays. 
Once you have the numpy data, you can transform them to torch.Tensors using torch.from_numpy(). 
Have a look at some examples from the <a href="https://docs.scipy.org/doc/scipy/reference/tutorial/io.html" rel="noopener nofollow ugc">scipy docs</a> on how to load the data.
  </blockquote>
</aside>
WGAN-GP lambda parameter

fllci: <p>Hi all!</p>
<p>I am building a WGAN-GP based CycleGAN model. On my dataset, when I use the lambda parameter for the gradient penalty as 1000, it gives a very good result (I mean it converges to zero). Yet in the original paper, the authors use lambda as 10. I am not sure whether 1000 is too much or not a a hyperparameter. But this gave me the best result so far.</p>
torch.nn.Sequential() vs. nn.Module()

pascalm: <p>Hello,</p>
<p>when defining a neural network, I usually make a class</p>
<pre><code class="lang-auto">class NeuralNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        ...

    def forward(self, inputs):
        ...
</code></pre>
<p>Now usually I define the things I need in the constructor and then built my NN inside the forward() but I mostly use sequential NN so I could also use torch.nn.Sequential(). I kind of like it better to read.</p>
<p>Was wondering what’s a good way to use torch.nn.Sequential() in the above? Like was good practice here? Would I just skrip defining my own class and make a function returning torch.nn.Sequential()?</p>
<p>Edit: Ah I guess they are kinda used together <a href="https://github.com/pytorch/vision/blob/main/torchvision/models/alexnet.py" class="inline-onebox" rel="noopener nofollow ugc">vision/alexnet.py at main · pytorch/vision · GitHub</a></p>
Pytorch tensor on custom calculation

Peter_Phan: <p>I have a function GetSimilarity (sentence1, sentence2).<br>
Now, I want to use this function in a long paragraph. Mean that I extract the paragraph into sentences, and want to measure the similarity between each sentence with the rest of the others by using my above function.<br>
For example, the paragraph contains 10 sentences, then I will use GetSimilarity(sentence1, sentence2) and GetSimilarity (sentence1, sentence3) …and so on.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f4d008aa2938efa9ffcc3d86bd554b9ed865ee9.png" alt="Capture" data-base62-sha1="91Z3wLaxEt9WcCVcC7IjBrKIWh3" width="575" height="76"></p>
<p>I am thinking about create a 2D square matrix which columns and rows are sentences as in the illustration above.<br>
May I ask can I use pytorch with GPU to do above calculation parallelly instead of using 2 nested loops and call the function for each pair of sentences?<br>
Thank you so much</p>
How to apply torchscript.pt in c++

zhangkun: <ol>
<li>i have trained a .pt model in python/pytorch. (yolov5)</li>
</ol>
<p>2)i have transform the .pt to torchscript.pt.</p>
<p>3)how to apply torchscript.pt in c++</p>
<p>my code is:<br>
auto outputs = model.forward({ data });<br>
auto pred1 = outputs.toList().get(0);</p>
<p>what is the following? please give me an example.</p>
How to allow in-place modification of saved tensors from forward pass in backward pass?

jasam-sheja: <p>I know saved tensors should not be modified because they could be used multiple times.<br>
However, when we save the output result of some layer, once the backward pass reaches that layer, there should not be any other layer that’d use it (right?) because the backward pass has already gone through them before. i.e., the output of A used in later B, C, etc., layers, so B, C, etc., should not modify it in place. Only A should be able to.</p>
<p>The problem PyTorch won’t allow any modification, period.</p>
<p>Why is this useful? We could reuse that Tensor for other calculations and save precious memory by doing it in place.</p>

ptrblck: <aside class="quote no-group" data-username="jasam-sheja" data-post="1" data-topic="137922">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jasam-sheja/48/44333_2.png" class="avatar"> jasam-sheja:</div>
<blockquote>
<p>The problem PyTorch won’t allow any modification, period.</p>
</blockquote>
</aside>
<p>I don’t think that’s true, as this code snippet shows that inplace manipulations are allowed after the computation graph was freed and thus no gradients can be (wrongly) calculated anymore:</p>
<pre data-code-wrap="python"><code class="lang-python"># setup
lin1 = nn.Linear(1, 1)
lin2 = nn.Linear(1, 1)
x = torch.randn(1, 1)

# disallowed since out1 is needed to calculate the gradients
out1 = lin1(x)
out2 = lin2(out1)
out1[0] += 1
out2.backward()

# manipulate afterwards
out1 = lin1(x)
out2 = lin2(out1)
out2.backward()
out1[0] += 1 # works
</code></pre>

jasam-sheja: <p>I see. What I want to do is something like this:</p>
<pre><code class="lang-auto">class AFunc(Function):
    @staticmethod
    def forward(ctx, x):
        res = function(x)
        ctx.save_for_backward(x, res)
        return res

    @staticmethod
    def backward(ctx, grad_output):
        x, res = ctx.saved_tensors
        res.inplace_op_(x)
        return res

x = torch.rand(1,1)
res = SomeFunc.apply(x)
out = use_res_func(res) 
out.backward()
</code></pre>
<p>So I want to modify during the backward pass while the computation graph is not (entirely?) freed.<br>
Since I want to modify <code>res</code> and not <code>x</code>, it’d be fine because <code>res</code> is created in the <code>forward</code> function, and when <code>backward</code> is reached, no other node in the remaining graph will use <code>res</code> anyhow.<br>
How do I tell PyTorch that it is okay to modify <code>res</code> in the <code>backward</code> function?</p>
Regression using Images and Tabular data

user26: <p>Hi,<br>
I am Trying to create an effeicient neural net and the best opzimizer + learning rate to take image and tabular data and get a rank (the target) that is a float, the model and optimizer are as following:</p>
<pre><code class="lang-python">
Model:

class MyModel7(nn.Module):
    def __init__(self):
        super(MyModel7, self).__init__()
        self.cnn = GoogleNet()# any conv net for image processing
        self.dropout = nn.Dropout(0.5)
        self.norm = nn.BatchNorm1d(250)
        # 12 tabular features
        self.fc1 = nn.Linear(12 + 1, 250)
        self.fc2 = nn.Linear(250,1)
        
    def forward(self, image, data):
        x1 = self.cnn(image)
        x2 = data
        
        x = torch.cat((x1, x2), dim=1)
        x = F.relu(self.fc1(x))
        #apply dropout
        x = self.dropout(x)
        # apply batchnorm
        x = self.norm(x)
        x = self.fc2(x)
        return x

optimizer:
optimizer = optim.Adam(model.parameters(), lr = 1e-3, weight_decay=1e-3)
</code></pre>
<p>the model is trained with an early stopping mechanism to ensure the best epoch parameters are saved</p>
<p>This model is performing ok but i want it to be as best as possible, what is the best way to approach this regression problem with the given data to minimize the MSE?</p>
Cannot re-initialize CUDA in forked subprocess on network.to(device) operation

Pascal_Niville: <p>Hello,</p>
<p>I am trying to implement the DistributedDataParallel class in my training code.<br>
The training code is a block in a larger block that I run to do the training and logging. Because the larger block runs twice when the multiprocess initialization method is set to  ‘spawn’ and rewriting my ‘main’ function would be too much work, I looked into forking the subprocess, so only the trainingsblock is run in parrallel.</p>
<p>The way I initialize the subprocesses now is (which i stole from <a href="https://pytorch.org/docs/stable/notes/multiprocessing.html" rel="noopener nofollow ugc">here</a>:</p>
<pre><code class="lang-auto">        import torch.multiprocessing as mp

        mp.set_start_method('fork', force=True)
        for rank in range(self.world_size):
            p = mp.Process(target=self.train, args=(rank,))
            p.start()
            processes.append(p)
        for p in processes:
            p.join()
</code></pre>
<p>Now the train function crashes on the “net.to(device)” line in the code sample below, with the error message:<br>
RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the ‘spawn’ start method</p>
<pre><code class="lang-auto">        self.net.to(rank)
        self.net = DistributedDataParallel(self.net, device_ids=[rank])
</code></pre>
<p>So far I’ve read that this can happen when something is already initialised on the cuda before the multiprocessing starts. But I can’t find where that would be in my code (I checked and removed all the .to(device) operations).</p>
<p>Other possible causes that I found could be related to the dataloader (num_workers to 0 and pin_memory), but placing the dataloader code after the network initialisation code, still gives the same error.</p>
<p>Is it possible to use the DistributedDataParallel class this way?<br>
Can the cuda be cleared before initialising the processes?</p>
<p>Thanks in advance</p>

wanchaol: <aside class="quote no-group" data-username="Pascal_Niville" data-post="1" data-topic="138090">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pascal_niville/48/31819_2.png" class="avatar"> Pascal_Niville:</div>
<blockquote>
<p>To use CUDA with multiprocessing, you must use the ‘spawn’ start method</p>
</blockquote>
</aside>
<p>thanks for posting <a class="mention" href="/u/pascal_niville">@Pascal_Niville</a>, this is a known issue for cuda runtime, you can see a related issue here <a href="https://github.com/pytorch/pytorch/issues/40403" class="inline-onebox" rel="noopener nofollow ugc">Cannot re-initialize CUDA in forked subprocess · Issue #40403 · pytorch/pytorch · GitHub</a>. The workarond is to use “spawn” instead of “fork” as suggested in the error.</p>
Exact range of acceptable/good rmse based on the range of target variable in regression problem using a DNN model

Mona_Jalal: <p>I am doing a regression problem on a target variable that its values range between 0 to 0.6 and highly skewed between 0 and 0.4.<br>
So, when I use rmse loss, I get the best rmse score as 0.21, is that very bad for the data? What would be a good range for rmse loss for this target variable prediction? and in a DNN-based regression, what are some ways you’d suggest to improve upon the rmse score?</p>
<p>I am doing a random train_test_split using sklearn with test data being 30% of the entire data.<br>
I wanted to do stratify on target variable, but many target values have only 1 and that method requires more than 2 in each bin.</p>
<p>Also, I only have 389 samples in my dataset.</p>
<p>Thanks a lot for any feedback.</p>