akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Running multiple modules in a ModuleList on different GPUs in parallel?

Sam_Lerman: <p>Is there a way to run the modules in a ModuleList in parallel on multiple GPUs? Their inputs are shaped differently and they don’t depend on one another.</p>

ptrblck: <p>Yes, if you are pushing different layers and their inputs to different GPUs, their execution will be asynchronous.</p>

Sam_Lerman: <p>Oh, but they’re in a for loop. Python will automatically proceed to the next iteration of the for loop?</p>

ptrblck: <p>Yes, CUDA kernels are launched by the CPU and are then executed asynchronously, which means the CPU can run ahead and execute other work, such as launching another CUDA kernel on another device.</p>

Sam_Lerman: <p>Okay, thanks. Hmm, would it work with DataParallel? Or do I manually have to map each iteration of the for loop to a specific GPU? All the tutorials I can find use DataParallel.</p>

ptrblck: <p><code>nn.DataParallel</code> works similar, but has communications added to its approach. <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">This blog post</a> explains its mechanism.</p>
Where to find python source code definition of nn.Conv2d

TigerVersusT: <p>Hello, I need to write a Conv2d module like nn.Conv2d but must inherit from nn.module instead of nn.Conv2d for some reason.  Can anyone tell me how to find the python source code definition of nn.Conv2d? I can’t find it on github. Thanks for your help.</p>

googlebot: <p>the top layer is in <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/conv.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/conv.py at master · pytorch/pytorch · GitHub</a></p>
How to deal with Sequential images?

Adwaitt: <p>Hello guys, I am tackling a challenge where I have a sequence of images, like in a video (starting from blank i.e all 0s to then having some content and then ending in 0s). For all these images I just have a single label. I wanted to ask, what is a good way to deal with such data? Can I do it like this</p>
<pre><code class="lang-auto">for i in all_images:
   y_pred = model.forward(i)
   loss = criterion(y_pred, y)
loss.backward()
</code></pre>

ptrblck: <p>Yes, your approach would technically work (just remove the <code>forward</code> call and use <code>model(i)</code> instead).<br>
I don’t know what model you are using, but in case it doesn’t use “states” from previous iterations (e.g. via RNNs), it would not learn any temporal dependency and would treat each input as a new sample.</p>

Adwaitt: <p>Thank you Peter You have always helped everyone <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
Shape of input must match shape of indices

Asya: <p>How to fix this error:</p>
<blockquote>
<p><code>RuntimeError: Shape of input must match shape of indices</code></p>
</blockquote>
<p>in max_unpool2d</p>
<blockquote>
<p>return torch._C._nn.max_unpool2d(input, indices, output_size)</p>
</blockquote>
<p>when I’m  trying to do unpooling on different size layer</p>

ptrblck: <p>As the error states the shape of the <code>input</code> and <code>indices</code> tensor have to match, which doesn’t seem to be the case in your script. Check the shapes via <code>print(input.shape)</code> and <code>print(indices.shape)</code> and make sure they are equal.</p>
Inverse of return_inverse :)

martinr: <p>Hi! torch.unique has <code>return_inverse</code> argument. When set to <code>True</code> it will return an index to re-construct the original tensor from the sorted one.</p>
<pre><code class="lang-auto">sorted_and_unique, idx = torch.unique(orig_input, 0, return_inverse=True)
# recontruct the original (same as orig_input)
original = sorted_and_unique[idx]
</code></pre>
<p>Would there be a way to get the reverse mapping <code>rev_idx</code> such that <code>orig_input[rev_idx]</code> would result in <code>sorted_and_unique</code>?</p>
<p>The use case is that I have two tensors <code>coords</code> (N by 3 tensor) and <code>features</code> (N by 125 tensor). I would like to remove duplicates and sort <code>coords</code> using <code>torch.unique</code> as above and then use <code>rev_idx</code> to <strong>select the same rows in same order</strong> from <code>feats</code>. In other words, <code>torch.unique</code> offers mapping from sorted tensor to original tensor, I would like to get a mapping from original tensor to sorted tensor.</p>

ptrblck: <p>I think <code>scatter_</code> should work:</p>
<pre><code class="lang-python">orig_input = torch.randint(10, 15, (10,))

sorted_and_unique, idx = torch.unique(orig_input, sorted=True, return_inverse=True, dim=0)
# recontruct the original (same as orig_input)
original = sorted_and_unique[idx]

print((orig_input == original).all())
&gt; tensor(True)

ret = torch.zeros_like(sorted_and_unique).scatter_(0, idx, orig_input)
print((ret == sorted_and_unique).all())
&gt; tensor(True)
</code></pre>
<p>However, note that the result might not be deterministic as you are writing to the same location in case your other tensor has different values, which should be mapped to the same index.</p>
Model not training after revisiting in a few weeks

Mrunalini_Ramnath: <p>I created a CNN implementation a few months ago and it worked great on different losses (Dice, CE, Focal). The iou and F1 scores would turn out great in just 3-5 epochs. However, I revisited it now a few weeks later and tried to re-train it on the same data but it’s not training. I initially tried to train it using Dice Loss (as it gave the best results) but it gave the no_grad error - something I had never faced before. I set requires_grad to True and it stopped giving the error but now it doesn’t really train. The values just hop around a bit but stay the same more or less over 10s of epochs. I tried it with the other losses too and they’re not training either. I have not changed the code base in any way. I have however done all this using colab. Would version changes or something along the lines contribute to this? Would really appreciate any help. I’ve been stuck on this for a while now and it’s incredibly frustrating.</p>

ptrblck: <aside class="quote no-group" data-username="Mrunalini_Ramnath" data-post="1" data-topic="133684">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrunalini_ramnath/48/42570_2.png" class="avatar"> Mrunalini_Ramnath:</div>
<blockquote>
<p>I set requires_grad to True and it stopped giving the error but now it doesn’t really train.</p>
</blockquote>
</aside>
<p>This sounds a bit dangerous as I guess you might have set the <code>requires_grad</code> attribute of the <code>loss</code> to <code>True</code> in order to avoid the “no grad error”?<br>
If so, your computation graph would still be detached and you would just mask the error.</p>
<p>Could you post the model definition as well as the loss function, which creates the issue, please?</p>
How to make an image into a Tensor without normalization on Android?

jinyoungsul: <p>I am making an application that compares two images to find the similarity.<br>
My model has been trained without image normalization.<br>
I want convert to image to Tensor without normalization.<br>
What should I do?</p>
<pre><code>
    public Tensor recognize(Bitmap image){

//        Tensor input =  TensorImageUtils.bitmapToFloat32Tensor(
//                image,
//                TensorImageUtils.TORCHVISION_NORM_MEAN_RGB,
//                TensorImageUtils.TORCHVISION_NORM_STD_RGB);

        float[] a = new float[]{1.0f, 1.0f, 1.0f};
        float[] b = new float[]{1.0f, 1.0f, 1.0f};
        Tensor input =  TensorImageUtils.bitmapToFloat32Tensor(
                image, a, b);


        Tensor output = module.forward(IValue.from(input)).toTensor();

        return output;
    }
</code>
</pre>

ptrblck: <p>Assuming the normalization subtracts the mean and divides by the stddev, you could set the mean to zeros and the stddev to ones, which would thus not change the tensor values.</p>
Finding mu and logvar

Asya: <p>How to find mean and log variance from the encoder latent space?</p>
<p>I receive <code>x = self.enc5(x) # torch.Size([75, 16, 101, 101])</code> from encoder</p>
<p>and i would like to get <code>mu</code> and <code>logvar</code> from <code>x</code> to pass it to:</p>
<pre><code class="lang-auto">    def reparameterize(self, mu, log_var):
        """
        :param mu: mean from the encoder's latent space
        :param log_var: log variance from the encoder's latent space
        """
        std = torch.exp(0.5 * log_var)  # standard deviation
        eps = torch.randn_like(std)  # `randn_like` as we need the same size
        sample = mu + (eps * std)  # sampling
        return sample
</code></pre>
<p>what is the right way to do so?</p>
<p>when I’m trying to do</p>
<pre><code class="lang-auto">        mu_logvar = x.view(x.shape[0], -1)
        mu = self.l1(mu_logvar)
        log_var = self.l2(mu_logvar)
        z = self.reparameterize(mu, log_var)
</code></pre>
<p>where</p>
<pre><code class="lang-auto">        self.l1 = nn.Linear(163216,500)
        self.l2 = nn.Linear(163216,500)
</code></pre>
<p>I get:</p>
<pre><code class="lang-auto">RuntimeError: Expected 4-dimensional input for 4-dimensional weight [16, 128, 4, 4], but got 2-dimensional input of size [75, 500] instead
</code></pre>
<p>on</p>
<pre><code class="lang-auto">x = F.relu(self.decoder1(z))
</code></pre>

ptrblck: <p>The output of <code>reparameterize</code> seems to be a 2D tensor in the shape <code>[75, 500]</code>, while <code>self.decoder</code> expects a 4D input and is most likely using a (transposed) conv layer.<br>
If that’s the case, <code>unsqueeze</code> the spatial dimensions and it might work:</p>
<pre><code class="lang-python"> x = F.relu(self.decoder1(z[:, :, None, None]))
</code></pre>
How to process an extremely uneven number of sequences?

335621: <p>I have an uneven training sequence for classification.<br>
For example, the image of the first category has 1000 tiles.<br>
The second category has 10 tiles.<br>
These tiles are the same size, e.g. 1024*1024.<br>
Is there the best training strategy?</p>
<p>The main problem is that I cannot pad all sequences to 1000 because of my limited GPU mem size.</p>
Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released a return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)

Joshua_Jenny_Sibbu: <p>I use google colab. colab was crash after getting this exception.<br>
I pass the single image to model with for loop it’s working fine. but if i use dataloader then it’s returning this exception and colab was crash.</p>
<pre><code class="lang-auto">class Dataset(torch.utils.data.Dataset):
  def __init__(self,inputImage):
    outLabel=pd.read_csv('/content/drive/MyDrive/image_labels.csv',index_col="image")
    self.outLabel=outLabel
    self.imagePath=imagePath
  
  def __len__(self):
    return (len(inputImage))
    
  def __getitem__(self,index):
    target_encoded=[]
    input_data=inputImage[index]    
    # path='/content/drive/MyDrive//images/',input_data
    # path=path[0]+path[1]
    # img=Image.open(path)
    #     # # images = images.to(device)
    #     # # labels = labels.to(device)
    #     # convert_tensor = transforms.ToTensor()
    # convert_tensor = transforms.Compose([transforms.Resize(size=(720,720)),transforms.ToTensor()])
    # img=convert_tensor(img)
    rows = self.outLabel.loc[[input_data]]
    dums=np.zeros((80,4))
    lst=[]
    for i in range(len(rows)):
      dums[i][0],dums[i][1],dums[i][2],dums[i][3]=rows['left'][i],rows['width'][i],rows['top'][i],rows['height'][i]
      # dums[i][1]=rows['width'][i]
      # dums[i][2]=rows['top'][i]
      # dums[i][3]=rows['height'][i]
    target_encoded=torch.tensor(dums)
    return input_data,target_encoded
</code></pre>

ptrblck: <p>The crash is most likely not coming from this warning, which should already be fixed in <a href="https://github.com/pytorch/pytorch/pull/60059">this PR</a>, so you might need to isolate the issue a bit more.</p>
Access tensor value through index value stored in another tensor

cbd: <p>For below code, How would I access the value of “x” for the index mentioned in “y”?<br>
Output should be [[1,9,17],[4,12,20]].</p>
<p>x shape is "torch.Size([3,1,8])  and y shape is "torch.Size([2, 2]).</p>
<pre><code class="lang-auto">import torch
x= torch.tensor([[[ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.]],

        [[ 8.,  9., 10., 11., 12., 13., 14., 15.]],

        [[16., 17., 18., 19., 20., 21., 22., 23.]]])


y=torch.tensor([[0,1],[0,4]])</code></pre>

KFrank: <p>Hi Cbd!</p>
<aside class="quote no-group" data-username="cbd" data-post="1" data-topic="133668" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/13edae/48.png" class="avatar"> cbd:</div>
<blockquote>
<p>Output should be [[1,9,17],[4,12,20]].</p>
</blockquote>
</aside>
<p>With a little fiddling around to get the indices in the right places, you<br>
can use indexing;</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; x = torch.tensor([[[ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.]],
...         [[ 8.,  9., 10., 11., 12., 13., 14., 15.]],
...         [[16., 17., 18., 19., 20., 21., 22., 23.]]])
&gt;&gt;&gt; y = torch.tensor([[0,1], [0,4]])
&gt;&gt;&gt; x[:, y[:, 0], y[:, 1]].T
tensor([[ 1.,  9., 17.],
        [ 4., 12., 20.]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

cbd: <p>I just want the answer just passing the variable y as indexing like “torch.function(x,y)” because in real there are so many values in tensor “y” for indexing in tensor “x”. Is it possible by using some pytorch function or trick.</p>
<p><img width="1" height="1" src="http://post.spmailtechn.com/q/QMYnC7JiXLT7WJuc31fN4w~~/AAIPyQA~/RgRjQEScPVcDc3BjQgphWxwRX2FWg1dfUhljaGludGFuX2dhdXJhdjhAeWFob28uY29tWAQAAAAA" alt=""></p>
<blockquote>
<p><img src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/45.png" title="KFrank" width="45" height="45"> | <a href="http://post.spmailtechn.com/f/a/qVWfxMId7dgvq0ltw00fBw~~/AAIPyQA~/RgRjQEScP0QkaHR0cHM6Ly9kaXNjdXNzLnB5dG9yY2gub3JnL3Uva2ZyYW5rVwNzcGNCCmFbHBFfYVaDV19SGWNoaW50YW5fZ2F1cmF2OEB5YWhvby5jb21YBAAAAAA~" rel="noopener nofollow ugc">KFrank</a> K. Frank<br>October 7 |</p>
<ul>
<li>| - |</li>
</ul>
</blockquote>
<p>Hi Cbd!</p>
<blockquote>
<p><img alt="" width="20" height="20" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/13edae/40.png">cbd:</p>
<p>Output should be [[1,9,17],[4,12,20]].</p>
</blockquote>
<p>With a little fiddling around to get the indices in the right places, you<br>
can use indexing;</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; x = torch.tensor([[[ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.]],
...         [[ 8.,  9., 10., 11., 12., 13., 14., 15.]],
...         [[16., 17., 18., 19., 20., 21., 22., 23.]]])
&gt;&gt;&gt; y = torch.tensor([[0,1], [0,4]])
&gt;&gt;&gt; x[:, y[:, 0], y[:, 1]].T
tensor([[ 1.,  9., 17.],
        [ 4., 12., 20.]])

</code></pre>
<p>Best.</p>
<p>K. Frank</p>
Tensor global max pooling and average

WangBo12: <p>hi, i am a beginner of pytorch.<br>
The problem is i have 16 tensors(each size is 14 * 14), and how could i use global max pooling and then calculate the average value of every 4 tensors, and return an output tensor with 4 * 1.<br>
Thanks.</p>

ptrblck: <p>You could use an adaptive pooling layer first and then calculate the average using a <code>view</code> on the result:</p>
<pre><code class="lang-python">x = torch.randn(16, 14, 14)
out = F.adaptive_max_pool2d(x.unsqueeze(0), output_size=1)

# Calculate result manually to compare results
out_manual = torch.stack([out[:, i:i+4].mean() for i in range(0, 16, 4)])

out = out.view(out.size(0), out.size(1)//4, -1)
out = out.mean(2)

print(torch.allclose(out_manual, out))
&gt; True
</code></pre>

WangBo12: <p><img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=6" title=":grinning:" class="emoji" alt=":grinning:">thanks. i figure it out</p>

vadimkantorov: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Do you know if <code>F.adaptive_avg_pool*</code> default to simply calling <code>torch.mean</code> if the output size is 1? (I want to use <code>adaptive_avg_pool*</code> for convenience but a bit hesitant because of potential overhead)</p>

ptrblck: <p>Yes, it should dispatch into <code>mean</code>.<br>
If I remember it correctly, this call was missing for <code>memory_format=torch.channels_last</code> workloads and was added before 1.7, so I would recommend to use the latest PyTorch version (as always <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"> ).</p>

synapse: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, is this the same as the TensorFlows’ GlobalMaxPooling?</p>

ptrblck: <p>Yes, I think that’s the case. Based on the docs in TF it seems you might need to add the <code>keedims=True</code> argument to get the same results and shape.</p>

synapse: <p>I see. What if I want a <code>2D array</code> as a result instead of the same shape (<code>3D</code> in my case)?<br>
Afaik, TF’s <code>GlobalMaxPooling</code> (if <code>keepdims=False</code>, which is so by default) outputs a <code>2D tensor</code>.</p>

ptrblck: <p>You could <code>squeeze</code> the spatial dimension via:</p>
<pre><code class="lang-python">pool = nn.AdaptiveMaxPool2d(output_size=1)

x = torch.randn(2, 3, 24, 24)
out =pool(x)
print(out.shape)
&gt; torch.Size([2, 3, 1, 1])
out = out.squeeze(3).squeeze(2)
print(out.shape)
&gt; torch.Size([2, 3])
</code></pre>
Preprocessing Dataloader Batches prior to sending them through to model

wolfman_numba1: <p>Hi there, (first time posting)</p>
<p>I am currently working on a university project where I’m aiming to use two different models on a dataset and combine the results during training. My question is regarding the preprocessing of my datasets. Currently my dataset is a variety of images.</p>
<p>For the first model, each image is converted to a numpy array that represents the information for each colour channel.</p>
<p>For the second model, facial elements of the image are extracted before being passed on to the model.</p>
<p>As you can see, each model requires different preprocessing before the data can be passed to each model. My current approach is to load the entire image dataset into a Dataloader and then while iterating through each batch, perform the preprocessing on the batches before passing them on to the models. I wanted to see if this is the correct approach.</p>
<p>Please let me know what you think. Thanks</p>

wolfman_numba1: <p>For anyone who finds this and is interested:<br>
My previous understanding was that <strong>getitem</strong> returned the data + the class. However, after playing around I realise we can return a list of data. My current approach will be to send a list, from getitem where the first element in the array will be the data related to my first model, and the second item in the array will be the corresponding data related to the second model.</p>
Finding the Number of Parameters given an Input Shape

ShikharAhuja: <p>Hello,</p>
<p>I wanted to write a variant of PReLU with one main distinction: it is always trainable for each channel-wise parameter instead of by layer.<br>
The official PReLU implementation requires the user to enter the nchannels if it is not layer by layer.<br>
Is there a way to discern that information given just the input tensor?</p>

ptrblck: <p>If I understand your use case correctly, you would like to use the first input to initialize the internal parameters of your custom activation module?<br>
If so, take a look at .g. <a href="https://github.com/pytorch/pytorch/blob/c0b1965f7c47f9a128fd00a9d835ea983f2fc4db/torch/nn/modules/linear.py#L197"><code>LazyLinear</code></a>, which does the same and try to adapt its usage to your module.</p>

ShikharAhuja: <p>Thanks for the info- I will check out the Lazy Linear function and see if it can be used for my purposes.</p>
About training a Faster RCNN with my loss functions

Sadegh_ab: <p>I want to train Faster RCNN with my loss function. I want to use RPN loss for training RPN Network and train ROI head and Classifier head with my loss functions. So, I have three loss functions here and, I want to train my network with these three losses simultaneously. How can I do this? is there a method in Pytorch for training ??</p>

ptrblck: <p>You could calculate the losses separately, accumulate them, and call <code>backward</code> on the final loss to calculate the gradients. Something like this would work:</p>
<pre><code class="lang-python">loss1 = criterion1(output1, target1)
loss2 = criterion2(output2, target2)
loss3 = criterion3(output3, target3)
loss = loss1 + loss2 + loss3
loss.backward()
</code></pre>
Epochs to be stored in different files

Rjacob: <p>Hi Folks,<br>
How would I be able to split my epochs to different files?<br>
For eg-<br>
If I am training my model for 20 epochs, I would like that at every five epochs, they get stored in 4 different files (.pt).<br>
How should I approach this?</p>
<p>Below is a code I am trying, any feedback would be appreciated-<br>
model = nn.Sequential(</p>
<p>nn.Linear(1, 30),</p>
<p>nn.Tanh(),</p>
<p>nn.Linear(30, 30),</p>
<p>nn.Tanh(),</p>
<p>nn.Linear(30, 1),</p>
<p>)</p>
<p>epochs=20<br>
<span class="hashtag">#Simple</span> network is a function I have defined to train my model, it works fine.<br>
results_pd = simple_network(model, loss_func, train_set, epochs=20,</p>
<p>test_set, checkpoint_file=‘model.pt’,</p>
<p>score_funcs={‘Acc’:accuracy_score})</p>

pchandrasekaran: <p>The point in the code where you are saving it could go something like this,</p>
<pre><code class="lang-auto">for e in range(epochs):
      .
      .
      .
      if e % 5 == 0:
         save_file_name = "{}_state.pt".format(e+1)
         torch.save(model_state_dict, save_file_name)
     .
     .
</code></pre>

Rjacob: <p>Thanks a lot, it helped!</p>
The size of a tensor a (1000) must match the size of a tensor b (100) at non-singleton dimension

alex_rachkov: <p>I’m a bit at a loss of why I am getting the following error when I run the following model:</p>
<pre><code class="lang-auto">class TestModel(torch.nn.Module):

    def __init__(self):
        super(TestModel, self).__init__()

        self.layer1 = torch.nn.Linear(1000, 100)
        self.relu = torch.nn.ReLU()
        self.layer2 = torch.nn.Linear(100,1)

    def forward(self,x):
        x - self.layer1(x)
        x = self.relu(x)
        x = self.layer2(x)
        return x
</code></pre>
<p>I get the error when I run the following line:</p>
<pre><code class="lang-auto">model = TestModel()
x = torch.randn(5,1000)
model(x)
</code></pre>
<p>And yet… when I test it through only the nn.linear function, I get no error, such as the following code:</p>
<pre><code class="lang-auto">linear = torch.nn.Linear(1000,100)
linear(x)
</code></pre>
<p>I have been staring at the monitor and scratching my head at this. Other similar errors I found online had to do with people using different shapes input vs target in the MSE loss, which is not the case here.</p>
<p>The error also goes away when I change layer1 to:</p>
<pre><code class="lang-auto">        self.layer1 = torch.nn.Linear(1000, 1000)

</code></pre>
<p>But I don’t want the output of the linear layer to be the same as the input. In fact, the size of the output units should not matter when it comes to the size of the input data (as we should be multiplying a 5x1000 matrix with a 1000x100 one).</p>

pchandrasekaran: <aside class="quote no-group" data-username="alex_rachkov" data-post="1" data-topic="133708">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alex_rachkov/48/42381_2.png" class="avatar"> alex_rachkov:</div>
<blockquote>
<p><code>x - self.layer1(x)</code></p>
</blockquote>
</aside>
<p>You have a typo here.</p>
Loss stuck. Model for speech-to-text system

Abdulrahman_Adel: <p>I’m trying to build a speech-to-text system my data is (4 - 10 seconds audio wave files) and their transcription (preprocessing steps are char-level encoding to transcription and extract mel-Spectrograms from audio files).<br>
this is my model architecture is ( a 3 conv1d layers with positional encoding to the audio file - embedding and positional encoding to encoded transcription and then use those as input to transformer model and lastly a dense layer)<br>
the loss function is cross entropy and optimizer is Adam.</p>
<p>the problem is that the loss is always stuck at some point it starts around 3.8 (I have 46 classes) and after some batches it decreases to (e.g.  2,8) and stuck their. it bounces around that value and never decrease again.<br>
I tried changing parameters of the model, I’ve changed the optimizer and learning rate always result the same problem.</p>
<p>I don’t understand what I’m doing wrong  <div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b855368f933c6f2ca97ce5d3db7ecad45d95975.png" data-download-href="https://discuss.pytorch.org/uploads/default/4b855368f933c6f2ca97ce5d3db7ecad45d95975" title="Screenshot 2021-09-28 192502"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4b855368f933c6f2ca97ce5d3db7ecad45d95975_2_690x296.png" alt="Screenshot 2021-09-28 192502" data-base62-sha1="aM5sustf4dEZ0Mx9YxWGb3ReJuZ" width="690" height="296" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4b855368f933c6f2ca97ce5d3db7ecad45d95975_2_690x296.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4b855368f933c6f2ca97ce5d3db7ecad45d95975_2_1035x444.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4b855368f933c6f2ca97ce5d3db7ecad45d95975_2_1380x592.png 2x" data-dominant-color="FDFAF9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-09-28 192502</span><span class="informations">1423×611 56.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

nateanl: <p>Hi <a class="mention" href="/u/abdulrahman_adel">@Abdulrahman_Adel</a>, what’s the format of your label? Usually the length of the text label is not the same as the feature sequence length, hence there is a CTC loss for addressing the issue. The output of the model contains “” symbol and it’s excluded when evaluating the word error rate.</p>
Quantize biases to quint8 for model deployment

dalnoguer: <p>Hello,</p>
<p>I have built a simple convolutional net and would like to quantize it for deployment.</p>
<p>Here is the code used for quantization.</p>
<pre><code class="lang-auto">def quantize_net(net):

    for module_name, module in net.named_children():

        if module_name in ['conv1', 'conv3_1', 'conv4_1', 'conv5_pa']:
            torch.quantization.fuse_modules(module, ['conv', 'bn', 'activation'], inplace=True)
        elif module_name in ['conv2', 'conv3_2', 'conv4_2']:
            for submodule_name, submodule in module.named_children():
                if submodule_name in ['depthwise', 'pointwise']:
                    torch.quantization.fuse_modules(submodule, ['conv', 'bn', 'activation'], inplace=True)
        if module_name in ['conv5_pb']:
            torch.quantization.fuse_modules(module, ['conv', 'bn'], inplace=True)

    net.qconfig = torch.quantization.get_default_qconfig('fbgemm')
    torch.quantization.prepare(net, inplace=True)

    net(torch.randint(low=0, high=255, size=(1000, 1, 40, 40), dtype=torch.float32))

    torch.quantization.convert(net, inplace=True)
</code></pre>
<p>All the convolution weights are quantized to quint8 but the biases stay in fp32. I assume that during inference biases are casted to int32 and added to the output of the conv operator. However, I would like to quantize the biases to quint8.</p>
<p>Is this supported at the moment? Is there a way I could achieve it?</p>
<p>Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

supriyar: <p>Hi <a class="mention" href="/u/dalnoguer">@dalnoguer</a></p>
<p>We currently don’t support quantizing biases to quint8. The main reason being our quantized backend (fbgemm) supports fp32 biases.</p>
<p>If you wish to update this to use quint8 bias, you’d have to write your own custom operator that accepts biases in this format along with the necessary kernel to do the computation in int8.</p>
Remove Tensor Column

r112: <p>Hello!</p>
<p>What would be the best way to remove every other column of this tensor:</p>
<p>tensor([[ 1,  2,  3,  4,  5],<br>
[ 6,  7,  8,  9, 10],<br>
[11, 12, 13, 14, 15]])</p>
<p>I am able to zero out every other row, but I am not sure how to remove those columns entirely instead of just setting them to zero.</p>
<p>Best,</p>

KFrank: <p>Hi Aryan!</p>
<aside class="quote no-group" data-username="r112" data-post="1" data-topic="133653" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/r112/48/29960_2.png" class="avatar"> r112:</div>
<blockquote>
<p>What would be the best way to remove every other column of this tensor:</p>
</blockquote>
</aside>
<p>Use tensor indexing:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.9.0'
&gt;&gt;&gt; t = torch.tensor([[ 1, 2, 3, 4, 5],
... [ 6, 7, 8, 9, 10],
... [11, 12, 13, 14, 15]])
&gt;&gt;&gt; t[:, [0, 2, 4]]
tensor([[ 1,  3,  5],
        [ 6,  8, 10],
        [11, 13, 15]])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

r112: <p>Hi Mr. Frank</p>
<p>What would be the best way to scale it up? Like taking out every other column for a 256 x 256 image? Every other row?</p>
<p>Thanks,<br>
Aryan</p>
Building a libjit with extern functions

iviarcio: <p>Hey everyone. I’m building a new backend for Glow called NMP (from NeuroMorphic Processor). It is an embedded device to make inferences from convolutional neural networks. So far I’ve used stubs functions (libjit_nmp) to develop the code generator, similar to LLVMIRGen. When I replace the stubs functions with function declarations, the getFunction() called inside the NMPIRGen module always returns nullptr. Note that the definitions of the functions are external (libobj_nmp) as they need to be compiled with GCC because of the device-specific toolchain that was only developed for GCC. Basically what I have in libjit folder now is a header file containing the function declarations and a few variables and constants defined in the source file. What I need is for the clang to keep the declarations in libjit_nmp.bc so that the NMPIRGen module can generate code in the main function to perform their calls. These references would then be resolved at link time with the lib_nmp.o library. Any ideas how to solve this?</p>

opti-mix: <p>Hi,</p>
<p>Yeah, the problem with the bitcode file generation is that Clang does not store LLVM IR function declarations if they are not really used, i.e. if these functions are not invoked inside the library.</p>
<p>Here are a couple possible work-arounds:</p>
<ol>
<li>
<p>You could add some empty bodies to your function declarations just to make Clang emit them into the BC file. Then, you can add some code at the very beginning of the your LLVMIRGen to remove the bodies from such functions (maybe based on their names or naming convention). This way they become declarations of external functions. Your code will compile just fine and you’ll be able to link them against the actual implementations from your GCC-compiled libraries.</p>
</li>
<li>
<p>Alternatively, you could at the very beginning of the your LLVMIRGen add the code to create declarations of your external functions using LLVM APIs for creating functions. The rest should just work.</p>
</li>
</ol>
<p>Please let me know if these solutions would work for you.</p>

iviarcio: <p>Thanks, Roman L for your feedback. I worked around this issue by creating wrappers for the external functions. This adds indirection but with clang optimization, it doesn’t affect performance.</p>
OpenMP not loading when using c++ via pybind

Karthik_Ganesan: <p>Hi, I am trying to implement a custom operator using C++ to work with PyTorch, via pybind. However, when I do ‘python setup.py install’ (i.e., build the C++ code with ninja), I get this warning:</p>
<pre><code class="lang-auto">/home/karthik/anaconda3/envs/pytorch-lib/lib/python3.9/site-packages/torch/include/ATen/ParallelOpenMP.h:87: warning: ignoring ‘#pragma omp parallel’ [-Wunknown-pragmas]
   87 | #pragma omp parallel for if ((end - begin) &gt;= grain_size)
</code></pre>
<p>The thing is I do want to use OpenMP with my code to speed it up. Is there some way to fix this so openMP gets included when I build?</p>
<p>And a related question, can I just add a “<span class="hashtag">#pragma</span> omp parallel for” in front of the loop or do I have to use the “at::parallel_for” function? Just in case the regular pragma causes any other issues with PyTorch. Thank you!</p>
Data loading error

fllci: <p>Hi all!</p>
<p>Could anybody help me with this error?</p>
<blockquote>
<pre><code>obj = super().__new__(cls, *args, **kwds)
</code></pre>
<p>TypeError: object.<strong>new</strong>() takes exactly one argument (the type to instantiate)</p>
</blockquote>
<p>This is the called class:</p>
<blockquote>
<p>class DamageAndUndamageDataset(Dataset):<br>
def <em>init</em>(self, csv_file, root_dir, transform=None):<br>
self.annotations = pd.read_csv(csv_file)<br>
self.root_dir = root_dir<br>
self.transform = transform</p>
<pre><code>def __len__(self):
    return len(self.annotations) #160

def __getitem__(self,index):
    img_path = os.path.join(self.root_dir, self.annotations.iloc[index,0])
    image = pd.read_csv(img_path)
    y_label = torch.tensor(int(self.annotations.iloc[index,1]))

    if self.transform:
        image = self.transform(image)
        
    return (image, y_label)
</code></pre>
</blockquote>
<p>And here is where I call the class:</p>
<blockquote>
<h1>Load data</h1>
<p>dataset = DamageAndUndamageDataset(csv_file= ‘labelsnormal’, root_dir = ‘datasetnormal’,<br>
transform = transforms.ToTensor())</p>
</blockquote>

ptrblck: <p>I can’t find any obvious issue in your code. Could you try to use some random tensors and post an executable code snippet, which would reproduce this issue, please?<br>
Also, which PyTorch version are you using?</p>

fllci: <p>Hi! Yesterday night I restarted my laptop and then it worked. I don’t know what caused the issue. Sometimes IT techniques “turn it off then on again” really works <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=10" title=":smiley:" class="emoji" alt=":smiley:"></p>
Including multiple and/or chained transforms for audio models

Mole_m7b5: <p>Does anyone have an efficient method of implementing and/or chaining together audio transforms within Pytorch Datasets or similar? I’m trying to make the dataset flexible enough that the user can input what transforms they want and whether any are chained together. torchvision has <code>Compose()</code> which allows chaining, so a possible solution might be to use <code>torch.nn.Sequential()</code> in a similar fashion. But that only allows for multiple transforms if they are chained together, not if you want to have two independent transforms to output two separate features i.e magnitude and phase.</p>
Implementation of Context Encoders in pytorch

ASHISH_MENON: <p>Has anyone implemented the paper “Context Encoders: Feature Learning by Inpainting” authored by Deepak Pathak, in pytorch??</p>

arjun_pukale: <p>I am also looking for the same especially that channel wise fully connected part</p>

mfarnas: <p>Wondering if any resource has been found on this that can be shared?</p>
Deterministic in torch.bmm

111576: <p>In the Doc of PyTorch 1.9.1(STABLE), torch.bmm(for batch mat matrix) evidently shown ‘deterministic’ for the sparse mat. But in the site-packages bmm is ‘def bmm(input: Tensor, mat2: Tensor, *, out: Optional[Tensor]=None)’ (from the Doc it should be ‘def bmm(input: Tensor, mat2: Tensor, *,deterministic=False, out: Optional[Tensor]=None)’). There is no function for batch sparse mats nor sparse mat-sparse mat matrix in the 1202 year.</p>
Torchscript inference is working on PC but is not working on Android

yousanke: <p>Hello All,</p>
<p>I have a torchscript that is working on PC ( PyTorch / libtorch ) for both trace mode and script mode, but when I run the same torchscript on my android application, it crashes without throwing explicit information.</p>
<blockquote>
<pre><code>Traceback of TorchScript, serialized code (most recent call last):
  File "code/__torch__/model_system.py", line 33, in forward
  def forward(self: __torch__.model_system.TransformerSystem,
    inputs: Tensor) -&gt; Tensor:
    spec = (self.preprop).forward(inputs, )
            ~~~~~~~~~~~~~~~~~~~~~ &lt;--- HERE
    outputs = (self.model).forward(spec, )
    outputs0 = torch.argmax(outputs)
  File "code/__torch__/torch/nn/modules/container/___torch_mangle_0.py", line 12, in forward
    _0 = getattr(self, "0")
    _1 = getattr(self, "1")
    input0 = (_0).forward(input, )
              ~~~~~~~~~~~ &lt;--- HERE
    return (_1).forward(input0, )
</code></pre>
</blockquote>
<p>Are there checkers or guidelines on how to land the torchscript to the Android from PC?</p>
How can I run the object detection model in the background

yusuf-5896: <p>I have a project developed with OpenCV C++ where I process images taken from the camera. Normally my project runs smoothly at 30fps. Now I am trying to add an object detection algorithm working with yolov5s lbtorch to develop the project. But the hardware I’m using can run the object detection model at around 10fps. But 1 fps is enough for my project.</p>
<p>My problem is when I add object detection model to my project it works as 8-9 fps. Can I run object detection in the background at 1fps and run normal operations at 30fps? I tried to briefly explain what I want to do in the picture below. How can I do that?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e86c21f133fdb288f8e37c917acca8ff928bbb9.png" data-download-href="https://discuss.pytorch.org/uploads/default/2e86c21f133fdb288f8e37c917acca8ff928bbb9" title="project"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/e/2e86c21f133fdb288f8e37c917acca8ff928bbb9.png" alt="project" data-base62-sha1="6DAGhmZkIdX4WjnCdIiui5c9xEJ" width="425" height="500" data-dominant-color="F8F8F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">project</span><span class="informations">453×532 6.33 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Hasintha_Hewage: <p>I have a similar question. Can you help me. I want to run trained yolov5s -“best.pt”  weight file in a normal CPU computer</p>

yusuf-5896: <p>I did it by creating a thread. As the thread completes the process, it continues to work by taking a new frame. I process by using intermediate outputs in the main thread.</p>
Export pth (without network code) to onnx

Ardeal: <p>Hi,</p>
<p>I would like to export pth file (without network code) to onnx, how to do that?</p>

ptrblck: <p>Could you describe that this <code>pth</code> file would contain? Would it be the <code>state_dict</code> or the model or any tensors?</p>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
Thank you!<br>
There are 2 cases:</p>
<ol>
<li>the pth file contains only state_dict,</li>
<li>the pth file contains the model.</li>
</ol>
<p>how to export those 2 kinds of pth to onnx?</p>

ptrblck: <ol>
<li>The <code>state_dict</code> contains all parameters and buffers and neither the model definition not any model code, so you can’t directly export it and would need to create a model instance first.</li>
<li>Assuming “contains the model” means a scripted model, you would again need to create the model object first and export it to ONNX as described <a href="https://pytorch.org/tutorials/advanced/super_resolution_with_onnxruntime.html">here</a>.</li>
</ol>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ，<br>
Thank you！</p>
<p>if the pth file contains only state_dict, is there a way to export the model to onnx?</p>
<p>If the pth file contains the model, I should export the model to onnx like this:</p>
<pre><code class="lang-python">model = torch.load(PATH)
model.eval()
torch.onnx.export(model, dummy_input, "alexnet.onnx", verbose=True, input_names=input_names, output_names=output_names)

</code></pre>

ptrblck: <ol>
<li>No, I don’t think so, since the <code>state_dict</code> only contains parameters and buffers and no model definition.</li>
<li>Yes, this might work assuming you can properly load the model.</li>
</ol>
Pretrained Torch vision models

OBouldjedri: <p>I was wondering on what size of ImageNet, the pretrained models of torch vision were pre-trained ?</p>
<p>ImageNet 1K<br>
or<br>
ImageNet21K<br>
or<br>
ImageNet22K</p>

ptrblck: <p>Based on this older post:</p><aside class="quote" data-post="20" data-topic="870">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/48/13_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/is-there-pretrained-cnn-e-g-resnet-for-cifar-10-or-cifar-100/870/20">Is there pretrained CNN (e.g. ResNet) for CIFAR-10 or CIFAR-100?</a> 
  </div>
  <blockquote>
    they’re all pre-trained on Imagenet-12
  </blockquote>
</aside>
<p>
they are pretrained on 2012 version of it and I would assume on the 1K classes version, since all pretrained classification models have 1000 output logits.</p>
Can i extract features from Mask RCNN?

mathwseg: <p>if I need to extract features using Pytorch from images by removing the last layer like in the VGG model but with the MASKRCNN model can I do that ? or it is just for object detection</p>

ptrblck: <p>It’s sometimes easier to replace the last layer with an <code>nn.Identity</code> module instead of trying to remove it in order to get the penultimate activations. I don’t know which layer you would like to remove, but this workflow might also work for you.</p>

mathwseg: <p>Thanks a lot for replying but excuse me  i tried to print the model using this code but didn’t work</p>
<pre><code class="lang-auto">class Identity(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return x
PATH = 'mask_rcnn_coco.h5'
model = modellib.MaskRCNN(    )
model.load_weights(PATH)
print(model)
</code></pre>
<p>the structure of MAskRcNN is<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/4/24634168341b5d12874355f050a38d41611dc3b3.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/24634168341b5d12874355f050a38d41611dc3b3" title="The-structure-of-the-Mask-R-CNN-architecture_W640"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/24634168341b5d12874355f050a38d41611dc3b3_2_288x250.jpeg" alt="The-structure-of-the-Mask-R-CNN-architecture_W640" data-base62-sha1="5bTQfWe2lB0aGdrsQR9GQN7KWsz" width="288" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/24634168341b5d12874355f050a38d41611dc3b3_2_288x250.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/24634168341b5d12874355f050a38d41611dc3b3_2_432x375.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/4/24634168341b5d12874355f050a38d41611dc3b3_2_576x500.jpeg 2x" data-dominant-color="D5DAE4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">The-structure-of-the-Mask-R-CNN-architecture_W640</span><span class="informations">640×555 52.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
i need to <code>nn.Identity</code> the classification layer to get the features that help in classify but don’t need to classify it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p><code>print(model)</code> should print out all the initialized submodules in their order. If you want to check the implementation you would have to check the source of <code>modellib.MaskRCNN</code> and probably the <code>forward</code> implementation to see which layer to replace.</p>
Find exact step size in Adam

Gantavya_Bhatt1: <p>Hi,</p>
<p>Suppose we have a neural network, and we optimize the corresponding loss using Adam. Let</p>
<pre><code class="lang-auto">\Delta_w = w_{t+1} - w_{t}
</code></pre>
<p>I am interested in finding out</p>
<pre><code class="lang-auto">\Delta_w / w_{t}
</code></pre>
<p>Is there a way to do that?</p>
Element 0 of tensors does not require grad and does not have a grad_fn at loss.backward()

Mrunalini_Ramnath: <p>Hey - I have a model that worked really well for segmentation. It would give pretty high iou scores in 5-6 epochs. I tried to re-train the model on another dataset 2 months later now and I’m now getting this error at loss.backward(). I have not changed the code in any way since then. I used a custom Dice Loss for the model. I set loss.requires_grad=True and it got rid of the error but now the model is more or less stuck at the same loss. I tried to re-train it on the same dataset but it’s giving the same issues. What could be causing this?</p>

tom: <p>Did you insert <code>torch.no_grad</code> by accident or cast through integer tensors somewhere?</p>
<p>One simple approach to debugging this kind of thing is to print various intermediates and check that they have the “grad_fn=…” information (or just print t.grad_fn or t.requires_grad for them ). The operation that takes things requiring gradients and outputting things that don’t will be the one that trips you up.</p>
<p>Best regards</p>
<p>Thomas</p>

Mrunalini_Ramnath: <p>I haven’t really made any changes to the code. I tried testing it out with BCEWithLogitsLoss() (which had also giveen pretty good results previously) in case there are any issues with just the Dice Loss but the loss isn’t really decreasing as such and it’s overall stuck at pretty low iou scores.</p>

tom: <p>So just setting requires_grad on the output as you mentioned on the first post will stop the backward at that tensor, so it would not update the neural network.<br>
So from your description you would want to find the operation breaking the autograd graph in order to get a meaningful backward pass again.</p>

Mrunalini_Ramnath: <p>Hey, I’ve gone over everything but I’m still unsure what could be breaking the autograd graph considering the exact same code worked perfectly a few weeks ago. I am running all of this on colab. Would that make any difference? (Would some automated version difference factor in? I really don’t understand what could be causing this as the exact same code worked well on the same dataset a while ago)</p>