akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Replace specific layer

Ahmad_Khan: <p>I  want to replace specific layers in pytorch for a given nn.Sequential.</p>
<p>Example: Replace maxpool  with average pool</p>

murtazanazir: <pre><code class="lang-python">&gt;&gt;&gt; model = nn.Sequential(nn.Conv2d(3,14,3),
                 nn.MaxPool2d((2,2)),
                 nn.Conv2d(14,28,3))
&gt;&gt;&gt; print(model)
Sequential(
  (0): Conv2d(3, 14, kernel_size=(3, 3), stride=(1, 1))
  (1): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)
  (2): Conv1d(1, 2, kernel_size=(4,), stride=(1,))
)
</code></pre>
<p>The MaxPool layer is numbered (1).</p>
<pre><code class="lang-python">&gt;&gt;&gt; print(m[1])
MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)
</code></pre>
<p>Now change it by simple assignment.</p>
<pre><code class="lang-python">&gt;&gt;&gt; m[1] = nn.AvgPool2d((2,2))
</code></pre>
<p>Check the model again.</p>
<pre><code class="lang-python">&gt;&gt;&gt; print(m)
Sequential(
  (0): Conv2d(3, 14, kernel_size=(3, 3), stride=(1, 1))
  (1): AvgPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0)
  (2): Conv1d(1, 2, kernel_size=(4,), stride=(1,))
)
</code></pre>
<p>Hope it helps.</p>
Use all the gpus for training

ziqipang: <p>I want to use multiple gpus for training. But as I have to do this training job on a cloud server, I cannot know the ids of gpu. However, the tutorial I have seen all need gpu ids to specify the usage. <a href="https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html</a> and <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#create-model-and-dataparallel" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#create-model-and-dataparallel</a></p>
<p>Therefore, I want to know if there is a method for me use all available devices?</p>

ptrblck: <p>You could use <code>torch.cuda.device_count()</code> to get the number of GPUs and then <code>torch.cuda.get_device_properties(idx)</code> to get the information of the device using <code>idx</code>.</p>

ziqipang: <p>Thank you! It works!</p>
Model training failed due to DataLoader worker (pid 12119) is killed by signal: Killed

Christos_Antoniou: <p>This is the error I get when i train my model. The error occured at the 10th epoch after running for 1 day. Is it a memmory issue? Cause i tried to use my multiple GPUs in parallel.What are your suggestions?</p>
<blockquote>
<pre><code>_error_if_any_worker_fails()
</code></pre>
<p>RuntimeError: DataLoader worker (pid 12119) is killed by signal: Killed.</p>
</blockquote>
<p>The screenshot of my model can be seen here:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/9/9c4e5b3ae07d1ef3ec40f6e72c2c5eb2306ce2ec.png" data-download-href="https://discuss.pytorch.org/uploads/default/9c4e5b3ae07d1ef3ec40f6e72c2c5eb2306ce2ec" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/9/9c4e5b3ae07d1ef3ec40f6e72c2c5eb2306ce2ec_2_690x388.png" alt="image" data-base62-sha1="miKmvSCs21wS99AZGefwWOHPJEM" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/9/9c4e5b3ae07d1ef3ec40f6e72c2c5eb2306ce2ec_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/2X/9/9c4e5b3ae07d1ef3ec40f6e72c2c5eb2306ce2ec_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/9/9c4e5b3ae07d1ef3ec40f6e72c2c5eb2306ce2ec_2_1380x776.png 2x" data-dominant-color="5E5E5E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×1080 406 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Sally_Goldin: <p>I am just getting my feet wet with PyTorch, and I am seeing the same behavior, with no GPU.</p>
<p>Running a CNN very similar to the 60 minute blitz tutorial, with two classes and approximately 12,000 training images, I get the following at epoch 87:</p>
<p>Traceback (most recent call last):<br>
File “training.py”, line 69, in <br>
loss.backward()<br>
File “/home/goldin/miniconda3/lib/python3.7/site-packages/torch/tensor.py”, line 166, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/goldin/miniconda3/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 99, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
File “/home/goldin/miniconda3/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py”, line 66, in handler<br>
_error_if_any_worker_fails()<br>
RuntimeError: DataLoader worker (pid 4452) is killed by signal: Killed.</p>
<p>This is replicable. It looks suspiciously like a memory leak since the first 86 epochs complete successfully.</p>
<p>The training is taking about 3 minutes per epoch, so it takes about four and a half hours for the problem to appear.</p>
<p>Is there a way to turn off the _error_if_any_worker_fails() flag?</p>
<p>Can you offer suggestions for a work around?</p>
<p>Should this be raised as an issue on GitHub?</p>
<p>Thank you!</p>
Padding applied even after giving padding=0 in nn.conv1d()

ruchitpatel: <p>I have created this model :</p>
<p>def count_parameters(model):<br>
return sum(p.numel() for p in model.parameters() if p.requires_grad)</p>
<p>a = nn.Module()<br>
a.l1 = nn.Conv1d(90000, 300, 2, padding=0)<br>
print(count_parameters(a))</p>
<p>It gives me output as 54000300. I think it should give only 53999700. The reason is, I’m not padding the sequence, and because the filter length is 2, our model should ignore the last input, as there is no any padded input node after last node. So technically, the output number of parameters should be (89999) * (2) * (300). But instead, the output is (90000)<em>(2)</em>(300). It does not make any difference if I pad the sequence or not. Even if I pad the sequence with padding=1 or padding=2, my answer remains unchanged, i.e. 54000300. Also in <a href="https://pytorch.org/docs/stable/nn.html#convolution-layers" rel="nofollow noopener">the</a> formula described in the documentation to calculate the output dimension, it does not take into account the padding. It always counts from 0 to Cin-1.<br>
It should count the sum by taking into account the filter_size. If it is more than 1, it should ignore the columns from the back accordingly. For example, if filter size is 2, then it can not take last column for convolution, since the sequence is not padded. Similarly, for filter size of 3, it should ignore last two columns because of the same reason.</p>

vdw: <p>I didn’t check your calculations, but maybe you can have a look at <a href="https://github.com/chrisvdweth/ml-toolkit/blob/master/pytorch/models/text/classifier/cnn.py" rel="nofollow noopener">my code here</a>: There are two methods <code>_calc_conv_output_size</code> and <code>_calc_maxpool_output_size</code> to calculate the sizes of the output of the conv layer and the max_pool layers – since I want to be flexible with the sequence lengths, kernel sizes, stride, etc.</p>
<p>Apart from that, <code>nn.Conv1d(90000, 300, 2, padding=0)</code> looks a bit surprising to me. From my understanding the set up is as follow:</p>
<pre><code>nn.Conv1d(in_channels=embedding_dim,
          out_channels=out_channels,
          kernel_size=conv_kernel_size,
          stride=self.common_conv_stride,
          padding=self.common_conv_padding)
</code></pre>
<p>So your embedding_dim is 90000? The whole purpose of word embeddings is to get a lower dimensional representation of words. Memory might also be a problem here.</p>
<p>Uh, wait a minute. The number of parameters of a layer is not the same as the number of outputs. i.e., the output dimension. For example, there are bias parameters in the layer that out not reflected in the output dimension. You’re comparing two different numbers here.</p>
Hello, i need help please i want to fine tune a pre trained model in my new data , i found a some data set " egohands " fine tuned in pytorch and i tried to follow exactly the same steps but my model doesn't detect anything , i can't found the error

leila30: <p>here is what i find like example<br>
<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html</a></p>

ptrblck: <p>If your training fails with your model and custom dataset, I would recommend to scale down the problem a bit and try to first overfit a small data sample.<br>
E.g. just take 10 samples out of your dataset and try to overfit these samples using your current code.<br>
If that works, you could try to scale up the use case again by using more data.</p>

leila30: <p>thank you for the answer acctualy i took juste 90 images and I done a mask for each images but i don’t know if my mask is correct</p>

leila30: <p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/khelloufelbakrilaaroussi/Mask-R-CNN-DocExplore" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/58788146?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/khelloufelbakrilaaroussi/Mask-R-CNN-DocExplore" target="_blank" rel="nofollow noopener">khelloufelbakrilaaroussi/Mask-R-CNN-DocExplore</a></h3>

<p>fine tune pre-trained model mask rcnn for DocExplore dataset - khelloufelbakrilaaroussi/Mask-R-CNN-DocExplore</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
please can you take a look on what i try to do ?</p>
How to share weights of parent custom class with the child custom class

Pratyush_Sinha: <pre><code class="lang-auto">class shared_feature():
        
    def __init__(self,out , inp):
        super(shared_feature, self).__init__()
        self.weights =torch.nn.Parameter(data =torch.Tensor(out , inp), \
                                                requires_grad=True)
        self.weights.data.uniform_(-1.0, 1.0)
        

class xtype_feature(nn.Module):
    
    def __init__(self,out , inp ):
        super(xtype_feature, self).__init__()
        self.feature =torch.nn.Parameter(data =torch.Tensor(out , inp), \
                                                requires_grad=True)
        self.weights.data.uniform_(-1.0, 1.0)

    def forward(self, x):
        #concatenation with weights of shared feature
        y= torch.cat(x, weights_from_shared_feature)
        return y 
        
</code></pre>
Tensor Cores with pytorch

hika_lika: <p>I have a RTX 2080 ti and i am totally new to CUDA. Is there is a benchmark to test the speedup of tensorcores compared to normal cuda cores with pytorch ? And is there is any good book recommendation for the topic ?</p>

albanD: <p>If I recall correctly, Tensor core only work for half precision floats.<br>
You can see the <a href="https://github.com/NVIDIA/apex">apex repo</a> from nvidia about how to do half precision floats.</p>
Create a pytorch type list for tensors of different sizes?

kaiseryet: <p>Hey guys, so I am trying to create a PyTorch type of list for a few tensors of different sizes but same dimensions; e.g.,</p>
<pre><code class="lang-auto">A = torch.randn(3,3)
B = torch.randn(5,5)
</code></pre>
<p>So I need to create a PyTorch type object like <code>[A, B]</code>, for the purpose of vectorized calculation (matrix multiplication, matrix inverse, …) happened later on (to each element in the list; i.e., <code>A,B,...</code>). How should I do this?</p>
<p>Further to this, is it possible to do this for a list of tensors of different dimensions?</p>

albanD: <p>Hi,</p>
<p>What you would want here is a NestedTensor. This is work in progress at the moment. You can follow the current status on this issue: <a href="https://github.com/pytorch/pytorch/issues/25032">https://github.com/pytorch/pytorch/issues/25032</a></p>

kaiseryet: <p>So nestedtensor is a separate package than torch? Interesting why torch doesn’t include it.</p>

ptrblck: <p>It’s a feature, which is currently being implemented. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

kaiseryet: <p>I just hope it can be implemented soon so that I don’t have to use nightly lol</p>

kaiseryet: <p>Speak of nestedtensor, just to confirm, it will be able to run on GPU right?</p>

albanD: <p>Yes it will.<br>
The current experimental repo <a href="https://github.com/pytorch/nestedtensor">here</a> should already support it.</p>
KL loss and log_softmax

barakb: <p>Hi, as mentioned in the docs, the student logits need to pass through a log_softmax, my question is <strong>why?</strong> log softmax doesn’t give us a distribution, is it because of the way pytorch implements kl loss?</p>
<p>Thanks!</p>

tom: <p><code>log_softmax</code> gives you logarithms of a distribution…</p>
<p>Expanding DL(P_target || P_prediction) = P_target * (log P_target - log P_prediction), we see that P_prediction only appears as a logarithm. Given that taking the exponential and then converting back is numerically somewhat unstable, it makes sense to avoid this step. That’s why KLDivLoss takes P_target and log P_prediction.</p>
<p>Best regards</p>
<p>Thomas</p>
TorchScript Model inference slow in python

RebirthT: <p>When testing if the prediction is correct after the model is transferring to script model, I found that the inference time is twice longer than pytorch model.</p>
<pre><code class="lang-auto"># pytorch 1.4, python 3.7
out1 = model(input) # forward time: 0.05 sec
ScriptModel = torch.jit.script(model, input)
out2 = ScriptModel(input) # forward time: 0.12 sec
</code></pre>

driazati: <p>The first pass does some extra work (details in the link below), if you warm up <code>ScriptModel</code> by calling it a few times first the time should improve.</p>
<aside class="quote quote-modified" data-post="1" data-topic="63209">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/c89c15/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/speed-of-custom-rnn-is-super-slow/63209">Speed of Custom RNN is SUPER SLOW</a> <a class="badge-category__wrapper " href="/c/jit/13"><span data-category-id="13" data-drop-close="true" class="badge-category " title="A category for TorchScript and the PyTorch JIT compiler"><span class="badge-category__name">jit</span></span></a>
  </div>
  <blockquote>
    Hi, 
Based on code here 

<a href="https://github.com/pytorch/pytorch/blob/master/benchmarks/fastrnns/custom_lstms.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/benchmarks/fastrnns/custom_lstms.py</a> 

I write an example to compare the cumputation capability of native lstm and custom lstm. 
But I found that the speed of custom lstm is 100 times slower than native lstm class. 
here is my test code: 
import torch.nn as nn
import time
from models.custom_lstm import LSTMLayer, LSTMCell, script_lstm, LSTMState

input_size = 1024
cell_size =2048
batch_size =20 
seq_len = 200 

native_lstm=n…
  </blockquote>
</aside>


RebirthT: <p>As the snippet below, script model actually get slower than average time 0.12sec in the first iteration in for loop as <a class="mention" href="/u/driazati">@driazati</a> mentioned above . However, after the first iteration, the rest is ~ 0.12 sec/img and is still twice longer than pytorch model.<br>
Put more information here: batch_size=1</p>
<pre><code class="lang-auto">pytorch 1.4, python 3.7
# pytorch model
for i in range(image_number):
    out1 = model(input) # forward time: 0.05 sec

#torchscript model
ScriptModel = torch.jit.script(model, input)
for i in range(image_number):
    out2 = ScriptModel(input) # forward time: 0.12 sec if i &gt; 0, 3 sec if i==0
</code></pre>

G.M: <p>If you are using the native lstm, it is normal that the native one is faster than touchscript. Also, according to <a href="https://pytorch.org/blog/optimizing-cuda-rnn-with-torchscript/#writing-custom-rnns" rel="nofollow noopener">here</a>, increasing the batch size can reduce the difference in performance. And according to my own experience, increasing the hidden size can also reduce the difference.</p>
Network dimension float16, the output is nan

slavavs: <p>If I make the network dimension float16, the output is nan (if epoch &gt; 0).</p>
<p>Forward pass is handled correctly (epoch = 0). After I do the backward, on the second epoch nan. Does anyone know why this is?</p>

slavavs: <p>If I do float32, then everything works correctly. But since my card is RTX, it is better that the network runs on float16.</p>

slavavs: <p>Adam(eps=1e-04) - I solved the problem</p>
Pytorch Batch Matrix Operation

Nick_8229: <p>I am trying to us torch.mm to do the following matrix operation,</p>
<p>If  matrix is a M * N tensor,  <code>batch</code>  is a N * B tensor, how can i achieve,<br>
In each batch, matrix @ batch_i, which gives M, and put the batch size together, the output tensor looks like M * B</p>
<p>There two questions here,<br>
1.To use torch.bmm, it seems need both matrix need be batch, but my first input is not<br>
2. The batch size need be the first dimension, while my batch size in the end</p>

b02202050: <p>As in the PyTorch document,<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/0/8076aa5990ea8c77bcc67200aad10d114cf65b38.png" data-download-href="https://discuss.pytorch.org/uploads/default/8076aa5990ea8c77bcc67200aad10d114cf65b38" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/8076aa5990ea8c77bcc67200aad10d114cf65b38_2_690x230.png" alt="image" data-base62-sha1="ikrm3pGIC1VzYLiHaPoTnErXEak" width="690" height="230" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/8076aa5990ea8c77bcc67200aad10d114cf65b38_2_690x230.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/0/8076aa5990ea8c77bcc67200aad10d114cf65b38_2_1035x345.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/0/8076aa5990ea8c77bcc67200aad10d114cf65b38.png 2x" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1044×348 26.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
torch.bmm should be used when both input are batched matrixes.<br>
But it seems like you are doing multiplication with a batch of vector by a constant matrix.<br>
If you really want to use bmm, you can make both of your inputs be the right format like the following.</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
M, N, B = 10, 12, 14
x = torch.rand(M, N)
y = torch.rand(N, B)

x_batch = x.repeat(B, 1, 1) # make x_batch be the shape of (B, M, N)
y_batch = y.T.unsqueeze(2) # make y_batch be the shape of (B, N, 1)
results = torch.bmm(x_batch, y_batch)
</code></pre>
Speed up torch.svd()

silvester: <p>I am doing some classification task for CIFAR10 with Pytorch and for each iteration I have to do some preprocessing on each batch before I would be able to feed forward to the model. Below is the code for the preprocessing part on each batch:</p>
<pre><code class="lang-auto"> S = torch.zeros((batch_size, C, H, W))
            for i in range(batch_size):
               Img = Batch[i, :, :, :]
               for c in range(C):                
                   U, _, V = torch.svd(Img[c])
                   S[i, c] = U[:, 0].view(-1, 1).matmul(V[:, 0].view(1, -1))
</code></pre>
<p>However, this calculation is very slow. Is there any way that I could speed up this code?</p>

LeviViana: <p>Here is a vectorized implementation:</p>
<pre><code class="lang-python">import torch

N, C, H, W = 8, 3, 50, 100

# Vectoried implementation
batch = torch.rand(N, C, H, W)
u, _, v = torch.svd(batch.view(-1, H, W))

s = u[:, :, 0, None].matmul(v[:, :, 0, None].transpose(2, 1)).view(N, C, H, W)

# Your current implementation
S = torch.zeros((N, C, H, W))
for i in range(N):
    Img = batch[i, :, :, :]
    for c in range(C):                
        U, _, V = torch.svd(Img[c])
        S[i, c] = U[:, 0].view(-1, 1).matmul(V[:, 0].view(1, -1))


# Verify the result
print(S.allclose(s)) # Prints True
</code></pre>

silvester: <p>Thanks for the code.<br>
I tried the vectorized version before and the improvement is marginal. Since I only need the first singular values, and its associated eigenvectors, is there  a way to avoid full svd computation?</p>
Training model with out-of-memory

kaiyu: <p>I have already trained a lstm model with epoch 30~.<br>
But when I add a CRF decode layer, it has a error with out-of-memory in epoch 6 or 7.<br>
I think it’s true. The model has many parameters, so I want to release some no-used parameters.<br>
what should I do? I don’t know which parameters is in the memory.<br>
All tensor.to(‘cuda’) ? or clone() ?<br>
Has pytorch the auto function to release the tensor?<br>
Thanks a lot</p>

Lornatang: <p>You can try to reduce the number of threads for DataLoader</p>

ptrblck: <p>Are you seeing the out of memory error on the GPU or are you running out of system RAM?<br>
In the former case could you check, if the memory usage is increasing in each iteration or epoch via <code>nvidia-smi</code>?<br>
If so, make sure to not save tensors attached to the computation graph, e.g. by appending the <code>loss</code> to a list without calling <code>loss.detach()</code>.</p>

kaiyu: <aside class="quote no-group" data-username="ptrblck" data-post="3" data-topic="69099" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Are you seeing the out of memory error on the GPU or are you running out of system RAM?<br>
In the former case could you check, if the memory usage is increasing in each iteration or epoch via <code>nvidia-smi</code> ?<br>
If so, make sure to not save tensors attached to the computation graph, e.g. by appending the <code>loss</code> to a list without calling <code>loss.detach()</code> .</p>
</blockquote>
</aside>
<p>thanks for your reply, actually it’s the error on the system RAM. How to solve this question? I pay attention to check the memory usage through ‘nvidia-smi’, but the GPU memory usage is very low, I have already put the device on Cuda.</p>

kaiyu: <aside class="quote no-group" data-username="Lornatang" data-post="2" data-topic="69099" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/3e96dc/48.png" class="avatar"> Lornatang:</div>
<blockquote>
<p>You can try to reduce the number of threads for DataLoader</p>
</blockquote>
</aside>
<p>I have already reduce the number of threads, and it’s working property on the original model.</p>
Retrain a pretrained model after adding Softmax layer

Ajinkya_Ambatwar: <p>Hi all, I wanted to know if it is needed to retrain the pretrained pytorch model if I wish to add a softmax layer at the end to get the class probabilities in case of classification problem?<br>
Eg.</p>
<pre><code class="lang-python">net = torchvision.models.resnet50(pretrained = True)
net.fc = nn.Sequential(nn.Linear(in_features=2048, out_features=1000))
net.fc = nn.Sequential(*list(net.fc)+[nn.Softmax(1)])
</code></pre>
<p>I am exporting this <strong>modified</strong> resnet50 pretrained model as onnx. On deployment on my browser, the class probabilities are being returned as 0% for top-4 classes. Previously using the default pretrained model it was coming as 1600% and so as there isn’t a context of class probabilities in the output layer in default pretrained resnet50.<br>
Is there a way I can introduce softmax without needing to train the network again?<br>
Thanks!!</p>

ptrblck: <p>You could simply add a softmax layer to the model if you need to see the probabilities.<br>
The output and the predicted class won’t change, as the max logit is also the max probability.<br>
To get the predicted class you could use <code>torch.aargmax(output, 1)</code> with <code>output</code> being the logits or probabilities.</p>
Purpose of SELayer

cnn_beginner: <p>Hi, can anyone tell me, what is the purpose of this Layer. I’m reading mobilenetV3 and GhostNet btw.</p>
<pre><code class="lang-auto">class SELayer(nn.Module):
    def __init__(self, channel, reduction=4):
        super(SELayer, self).__init__()
        self.avg_pool = nn.AdaptiveAvgPool2d(1)
        self.fc = nn.Sequential(
                nn.Linear(channel, channel // reduction),
                nn.ReLU(inplace=True),
                nn.Linear(channel // reduction, channel),
            )

    def forward(self, x):
        b, c, _, _ = x.size()
        y = self.avg_pool(x).view(b, c)
        print(y.shape)
        y = self.fc(y).view(b, c, 1, 1)
        print(y.shape)
        y = torch.clamp(y, 0, 1)
        print(y.shape)
        return x * y
</code></pre>
<p>Is it some kind of special operation for weighting feature?<br>
Thank you,</p>

ptrblck: <p>It looks like the <a href="https://arxiv.org/pdf/1709.01507.pdf">Squeeze-and-Excitation block</a>.</p>
Video classification networks

Subhankar_Ghosh: <p><a href="https://pytorch.org/docs/stable/torchvision/models.html#video-classification" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/torchvision/models.html#video-classification</a></p>
<p>I am going through the tutorial in the link above. It’s mentioned:</p>
<blockquote>
<p>We provide models for action recognition pre-trained on Kinetics-400. They have all been trained with the scripts provided in <code>references/video_classification</code> .</p>
</blockquote>
<p>I’m unable to find where the scripts are. Can any of you help me out?</p>
<p>Thanks!</p>

ptrblck: <p>You can find the scripts <a href="https://github.com/pytorch/vision/tree/master/references/video_classification">here</a>.</p>

Subhankar_Ghosh: <p>Thanks a lot for the help!</p>
Pytorch Wrapper v1.1

jkoutsikakis: <p>Hey all,</p>
<p>PyTorch Wrapper version 1.1 is out!</p>
<p>New Features:</p>
<ul>
<li>Samplers for smart batching based on text length for faster training.</li>
<li>Loss and Evaluation wrappers for token prediction tasks.</li>
<li>New nn.modules for attention based models.</li>
<li>Support for multi GPU training / evaluation / prediction.</li>
<li>Verbose argument in system’s methods.</li>
<li>Examples using Transformer based models like BERT for text classification.</li>
</ul>
<p>Check it out in the following links:</p>
<p>install with: pip install pytorch-wrapper<br>
GitHub: <a href="https://github.com/jkoutsikakis/pytorch-wrapper" rel="nofollow noopener">https://github.com/jkoutsikakis/pytorch-wrapper</a><br>
docs: <a href="https://pytorch-wrapper.readthedocs.io/en/latest/" rel="nofollow noopener">https://pytorch-wrapper.readthedocs.io/en/latest/</a><br>
examples: <a href="https://github.com/jkoutsikakis/pytorch-wrapper/tree/master/examples" rel="nofollow noopener">https://github.com/jkoutsikakis/pytorch-wrapper/tree/master/examples</a></p>
<p><span class="hashtag">#DeepLearning</span> <span class="hashtag">#PyTorch</span> <span class="hashtag">#NeuralNetworks</span> <span class="hashtag">#MachineLearning</span> <span class="hashtag">#DataScience</span> <span class="hashtag">#python</span> <span class="hashtag">#tensor</span></p>
Saving model that contains list of module

bedrick: <p>this is my model:</p>
<pre><code>def __init__(self, features, num_classes=8, init_weights=True):
    super(VGG, self).__init__()
    self.features = features
    self.roiNum = 6
    self.roi_pool = _RoIPooling(5,5,1.0/32.0)
    self.rois_classifiers =   []
    classifier = nn.Sequential(
        nn.Linear(512 * 5 * 5, 512),
        nn.ReLU(True),
        nn.Dropout(),
        nn.Linear(512, 512),
        nn.ReLU(True),
        nn.Dropout(),
        nn.Linear(512, num_classes),
    ).cuda()
    for i in range(self.roiNum):
      self.rois_classifiers.append(copy.deepcopy(classifier))

    if init_weights:
      self._initialize_weights()
</code></pre>
<p>when i saved the model, it only saves  self.feautures and self.roi_pool parameters. self.rois_classifiers parameters are not saved. Also when i print the model, again it just printed features and roi_pool layers, not rois_classifier layers. However, when i print model.rois_classifiers, everything is fine.</p>
<p>What is the problem here?</p>

ptrblck: <p>You would have to use <code>nn.ModuleList</code> instead of a Python list for <code>self.roi_classifiers</code> to properly register them as submodules.<br>
Inside a module, all directly assigned <code>nn.Modules</code> and <code>nn.Parameters</code> will be registered.<br>
Tensors, which should not be trained, can be registered as buffers via <code>self.register_buffer</code>.</p>
Do optimizers minimize or maximize?

cruzas: <p>Hi everyone,</p>
<p>I believe this has been answered implicitly but not explicitly elsewhere.</p>
<p>Do PyTorch optimizers minimize or maximize a loss function?</p>
<p>Thank you.</p>

ptrblck: <p>Optimizers subtract the gradient of all passed parameters using their <code>.grad</code> attribute as seen <a href="https://github.com/pytorch/pytorch/blob/6249d7302b7277864ed0ade93f58d88ee0cd3aa8/torch/optim/sgd.py#L110">here</a>. Thus you would minimize the loss using gradient descent.</p>
Way of modeling multi-dimensional output?

George3d6: <p>Assume I think a reasonable shape for my output is n <code>m x m</code> matrices, maybe I believe my loss would be best modeled in terms of these matrices (e.g. I’d prefer to predict n/2 of them correctly, rather than predict every 2nd element correctly but have 0 overall correct matrices).</p>
<p>I would just have a linear layer of size: n * m * m as the output from my network, and then construct a custom loss function I apply to each m*m chunk of my output, at least that was my first though.</p>
<p>However, I assume there might be a better way to approach such a problem. So I’m curios if anyone has any ideas of experience doing this ?</p>
<p>If anyone knows how to do this for a more general case (e.g. something that would apply to a list of arrays of arbitrary dimensions, rather than to a list of 2-dimensional arrays as per my example) I’d be even more curios to know that.</p>
<p>Are there any built-in loss functions that could be used for these type of problem or would I be better off constructing one from scratch ?</p>
Getting gradients of a neural network w.r.t. to batched inputs

rpfeynman: <p>Hi everyone! I’m trying to implement a network where I can define derivatives of its output w.r.t. the inputs.</p>
<p>To make things clearer: I’m using the neural network to approximate a scalar-valued function <img src="https://latex.codecogs.com/png.latex?f%28%5Cmathbf%7Bx%7D%29%3A%20%5Cmathbb%7BR%7D%5EN%20%5Cto%20%5Cmathbb%7BR%7D" alt width="118" height="20"></p>
<p>Since the output is a scalar, I can get the gradient  w.r.t a given <strong>single</strong> input simply using</p>
<pre><code class="lang-python">from torch.autograd import grad

x.requires_grad = True
u = model(x)
u_x = grad( u, x, retain_graph=True, 
            create_graph=True, 
            allow_unused=False)[0]
</code></pre>
<p>And this is very nice because I can use the gradient in my loss, for example by building up a regularization term, and I’m able to backpropagate through its graph.</p>
<p>But how can I extend this beyond this simple example? More specifically, I have the following couple of related questions:</p>
<ol>
<li>Is it possible to extend this beyond the first derivative? Assume that I want to calculate the Laplacian of <code>f(x)</code>, how can I do that? I <strong>can’t</strong> run again the <code>grad</code> function like so</li>
</ol>
<pre><code class="lang-python">u_xx = grad( u_x, x, retain_graph=True, 
             create_graph=True, 
             allow_unused=False)[0]
</code></pre>
<p>since <code>u_x</code> is not a scalar and therefore the gradient doesn’t exist. What I would like to have is practically the diagonal of the Jacobian matrix, but how can I get it?</p>
<ol start="2">
<li>Can I use batched inputs? Once more, with batched inputs I can’t run the <code>grad</code> function on the output because it will be a vector. Again, this is equivalent to ask for the diagonal of the Jacobian w.r.t. the inputs.</li>
</ol>

rpfeynman: <p>Bumping this up, since I haven’t seen a solution yet (especially for pt. 2)</p>
Function 'PowBackward0' returned nan values in its 0th output

MalumaDev: <p>Hi,<br>
by using <code>torch.autograd.set_detect_anomaly(True)</code> Pytorch return this error:</p>
<pre><code class="lang-auto">Function 'PowBackward0' returned nan values in its 0th output.
</code></pre>
<p>more specifically when I use <code>pow</code>:</p>
<pre><code class="lang-auto">_x = torch.where(_x &lt;= 0.008856, 9.033 * _x, 1.16 * (_x+epsilon).pow(1 / 3) - 0.16)
</code></pre>
<p>Someone has some ideas of how to avoid this?</p>

albanD: <p>Hi,</p>
<p>Is it possible for <code>_x</code> to be negative?<br>
Can you double check that <code>_x+epsilon</code> &gt; 0 ?</p>

MalumaDev: <p><code>_x</code> can be negative but the pow is applied only for <code>_x &gt; 0.008856</code>. Right?</p>

albanD: <p>I’m afraid, you might see the same problem as <a href="https://github.com/pytorch/pytorch/issues/15506" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/15506</a></p>
<p>Can you try to run <code>_x = torch.where(_x &lt;= 0.008856, 9.033 * _x, 1.16 * (_x.abs()+epsilon).pow(1 / 3) - 0.16)</code><br>
This should give the same value but avoid any nan gradients.</p>

MalumaDev: <p>You are right. Thanks!!</p>
Loss dosn't decrase at all when using lstm

SimTorch: <p>Hi,<br>
I implemented a classification model to fix a Five-class problem, but I got a problem and the work stopped, the output of net, everything in batch is 1(class No2), while the real label is 0,1,2,3,4.<br>
why this happen? and how? Thank u, everyone!!<br>
Here is my code:</p>
<blockquote>
<p>def forward(self, input):</p>
</blockquote>
<blockquote>
<pre><code>    if input is None or input.size()[-1] != self.input_size:
        return None
    out1,_= self.lstm(input,self.lstm_states)  
    lstmoutsize = self.hidden_size * 1 if self.bidirectional == 0 else 2
    lstmout = out1[:, -1, :]  
    lstmout = lstmout.view([-1,lstmoutsize]) # lstmout = batchsize * lstmoutsize
    outFcLstm = self.fcLstm(lstmout)             
    self.reset_hidden_states()

    return outFcLstm 
</code></pre>
</blockquote>
<p>main:</p>
<blockquote>
<pre><code>    criterion = T.nn.CrossEntropyLoss()
    lr = opt.lr
    optimizer = T.optim.Adam(rnn.parameters(), lr=lr, weight_decay=opt.weight_decay)

    while True:
        batch = self.queBatchSample.get()
        if batch is None:  
            break
        optimizer.zero_grad()
        output= rnn(batchData)
        if output is None:
            continue
        loss = criterion(output, batchLabel)
        loss.backward(retain_graph=True)
        optimizer.step()**
</code></pre>
</blockquote>
Weight type float16

slavavs: <p>I want to feed the model type float16. But I get the error RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.cuda.FloatTensor) should be the same. How do I set weight type float16?</p>

tom: <p>Did you try <code>m.to(dtype=torch.half)</code>?</p>
<p>Best regards</p>
<p>Thomas</p>
Faster R-CNN with FPN

sigma_x: <p>FPN outputs 4 layers of different sizes. How are RPN and RoI connected to them? RPN should output a set of anchors/layer, so 4 sets of anchors. Is RoI connected to just 1 of them to get the pooled feature map, or the corresponding feature map? E.g. for an anchor from pyramid layer 0, <code>layer_blocks</code> 0 is used?<br>
Thanks<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f6e9dd7f582a7216eb2b28666b6072dbd147cd70.png" data-download-href="https://discuss.pytorch.org/uploads/default/f6e9dd7f582a7216eb2b28666b6072dbd147cd70" title="fpn"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f6e9dd7f582a7216eb2b28666b6072dbd147cd70_2_690x255.png" alt="fpn" data-base62-sha1="zeivIPA8FrjQ4l6NDClIcwnj6Lu" width="690" height="255" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/6/f6e9dd7f582a7216eb2b28666b6072dbd147cd70_2_690x255.png, https://discuss.pytorch.org/uploads/default/original/3X/f/6/f6e9dd7f582a7216eb2b28666b6072dbd147cd70.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/6/f6e9dd7f582a7216eb2b28666b6072dbd147cd70.png 2x" data-dominant-color="38192E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">fpn</span><span class="informations">846×313 46.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
RPN+ROI architectiure (Faster R-CNN)

sigma_x: <p>I have a question about the architecture of RPN and ROIALign in Faster/Mask R-CNN (see the image).</p>
<p>For each anchor in RPN there are two ‘heads’: 4 bbox values and 1 confidence value that are compared to the labels (offsets for bbox and 1/0 for overlap). In the pytorch implementation though, <code>cls_logits</code> have 3 channels, and <code>bbox_pred</code> 3x4=12. I don’t quite understand, what these 3 channels mean. There should be 1 and 4, as I understand.</p>
<p>In RoIPooling, there are two heads, <code>cls_score</code> with the number of classes (91), and <code>bbox_pred</code> with 4*91=364. I don’t quite understand, why we need bbox/class? We have a softmax prediction, so only 1 class can exist in this anchor, so we need 1 bbox, where do the other 360 come from? And what does the label look like? Offsets for 4 correct values and 0s otherwise?</p>
<p>Thanks!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/1/b13d8438e0602af684525c17718ed640c85a8de0.png" data-download-href="https://discuss.pytorch.org/uploads/default/b13d8438e0602af684525c17718ed640c85a8de0" title="rpn_roi_structure"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b13d8438e0602af684525c17718ed640c85a8de0_2_690x467.png" alt="rpn_roi_structure" data-base62-sha1="phWjgvQWABaaQLk4YfKDUia6zUk" width="690" height="467" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/1/b13d8438e0602af684525c17718ed640c85a8de0_2_690x467.png, https://discuss.pytorch.org/uploads/default/original/3X/b/1/b13d8438e0602af684525c17718ed640c85a8de0.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/1/b13d8438e0602af684525c17718ed640c85a8de0.png 2x" data-dominant-color="37192D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">rpn_roi_structure</span><span class="informations">934×633 112 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
Inflating the tensor up to a certain size

kaiseryet: <p>Let’s say if I have a tensor of size <code>(a,b,c_1)</code>, and now I wish to inflate it to <code>(a,b,c_2)</code> for some <code>c_2 &gt; c_1</code>, is there a function to do this? Note that I am NOT changing the number of dimensions of the tensor; instead, I am just increasing one or more dimensions of the tensor. Thanks.</p>

ptrblck: <p>You could use <code>tensor.expand</code>, if you want to increase a singleton dimension with a view, <code>tensor.repeat</code> to repeat the values in this dimension, or <code>torch.cat</code> to concatenate different tensors and thus increasing the size in this dimension.</p>
Model is getting slower on CPU when I increased the number of threads

mundher: <p>During Inference, I noticed PyTorch is utilizing only 50% of my CPU threads. Then I tried to increase the number of threads to 10, 12, and 16 (my CPU has 16 threads) by calling set_num_threads. Although setting the number of threads to 16 makes all threads busy, the model becomes slower. even when no other processes are running.</p>
<p>Is it a bug?<br>
Could you please explain why I cannot utilize all my threads?</p>

ptrblck: <p>Too many threads might decrease the overall performance. E.g. if the workload for each thread is too small, you’ll suddenly notice the overhead of each thread.</p>
Multiple Outputs of a NN

BuVu: <p>Dear all,</p>
<p>Currently I am building a neural net to estimate the uncertainty in a regression, which is performed by the neural net. I will use a custom loss to update the weights of the neurons. The custom loss consits of two values, which are the outputs of the neural net. For now I am using nn.MSEloss() the perform the regression. I’m quite unsure how exactly i can implement two outputs. I would be thankful for any help regrading this issue. My Code is displayed below</p>
<pre><code class="lang-auto">class NN(nn.Module):
    def __init__(self, n_feature, n_hidden, n_output):
        super(NN, self).__init__()
        self.hidden = nn.Linear(n_feature, n_hidden)
        self.predict = nn.Linear(n_hidden, n_output)
        #probably here i can define the outputs

    def forward(self, x):
        x = torch.sigmoid(self.hidden(x))     
        x = self.predict(x)     
       #define the activation func for the outputs
        return x

net = NN(n_feature=1, n_hidden=10, n_output=1)

#create the optimizer
optimizer = torch.optim.SGD(net.parameters(), lr=0.2)

def MyLoss():
    
    #takes outputs of the neural net to compute 

   return None
    

n_iter = 200

#training loop
for i in range(n_iter):
    prediction = net(x)     # input x and predict based on x

    loss = My_Loss()    

    optimizer.zero_grad()   # zero the gradient buffers
    loss.backward()         # backpropagation, compute gradients
    optimizer.step()        # apply gradients   
.
.
.
</code></pre>

ptrblck: <p>You can simply return two tensors in the <code>forward</code> method of your model.<br>
E.g. using your model:</p>
<pre><code class="lang-python">class NN(nn.Module):
    def __init__(self, n_feature, n_hidden, n_output):
        super(NN, self).__init__()
        self.hidden = nn.Linear(n_feature, n_hidden)
        self.predict = nn.Linear(n_hidden, n_output)
        #probably here i can define the outputs

    def forward(self, x):
        x1 = torch.sigmoid(self.hidden(x))     
        x2 = self.predict(x1)     
       
        return x1, x2
</code></pre>
<p>would return now the intermediate activation as well as the last output.</p>
Difference between nn.BatchNorm2d() and transforms.Normalize((0.1307,), (0.3081,))

Sanjayvarma11: <p>Hi all,<br>
Here i am trying to improve accuracy of MNIST dataset to 99.14%.So i came across  nn.BatchNorm2d()  which is a regularization technique to improve accuracy by decreasing noises.But in the below code while downloading MNIST datset i applied a function in torch vision named  transforms.Normalize((0.1307,), (0.3081,))</p>
<pre><code class="lang-auto">train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('../data', train=True, download=True,
                    transform=transforms.Compose([
                        transforms.ToTensor(),
                        transforms.Normalize((0.1307,), (0.3081,))
                    ])),
</code></pre>
<p>Here in last line i am normalising each tensor.</p>
<ol>
<li>so what is the difference between both of them??<br>
2)should i use nn.BatchNorm2d()  or  it is equal to transforms.Normalize().<br>
3)So eventhough i use it does it make my image much darker or not??</li>
</ol>

ptrblck: <ol>
<li>
<p><code>transforms.Normalize</code> is used during preprocessing the input tensors not inside the model. BatchNorm layers have running estimates to normalize the activations of the previous layer as well as affine trainable parameters in the default setup.</p>
</li>
<li>
<p>Both approaches try to normalize tensors, but are used at different places and BatchNorm layers will estimate the running stats of the activations.</p>
</li>
<li>
<p>The brightness of your images might be just a visualization issue. E.g. if you normalize your input images to have a zero mean and a unit variance, the library you are using to plot these images might try to map these values to the colormap it is using.</p>
</li>
</ol>
Train a network to tolerate undesired weight shift during inference

xsun377: <p>Hi,</p>
<p>Is there anything could be done in the training to help the network be more resilient to undesired weight shift during inference?</p>
<p>For instance, suppose we train a simple MLP 784-512-10 and load the trained weights to some hardware, but the weight values are drifting down due to the reliability issue of memory components, thus degrading the accuracy. Assuming a known decreasing rate (v) of weight values, Is there any way to enforce a  safety margin of weights that can retain a good accuracy to a certain time (e.g., 1 year) ?</p>
<p>Any inputs are appreciated! Thanks!</p>

ptrblck: <p>Could you explain this (interesting) use case a bit more?<br>
Why would weight values drift down, if your memory is unreliable?<br>
Wouldn’t this introduce random errors in the memory and thus corrupting the values in an unpredictable way?</p>
Why can the dimensions of input and target be different when using this class of torch.nn.CrossEntropyLoss?

SangYC: <p>Why can the dimensions of input and target be different when using this class of torch.nn.CrossEntropyLoss? I remember that one_hot processing was needed when using keras. This’s why?</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> expects the target as a <code>LongTensor</code> containing the class indices rather than a one-hot encoded target.<br>
Since you are indexing the logits for the target of the corresponding sample, you can use the class indices directly.<br>
Have a look at the <a href="https://pytorch.org/docs/stable/nn.html?highlight=crossentropyloss#torch.nn.CrossEntropyLoss">docs</a> for more information.</p>

SangYC: <p>Thanks for your help.</p>