akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Nan gradient after loss.backward() because of Pow()

Suyoung_Park: <pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.w = nn.Parameter(torch.randn(1,1,3,3))
        self.myParam = nn.Parameter(torch.rand(1))

    def myfun(self,w):
        tempVar = torch.where( abs(self.w) &gt;=0.3, self.w**self.myParam, self.w)
        return tempVar

    def  forward(self,x):
        w = self.myfun(self.w)
        return F.conv2d(input, w)

net = Net()

input = torch.randn(1,1,50,50)
target = torch.ones(1,1,48,48)
loss_fn = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.Adam(net.parameters(),lr=0.0001)


for i in range(1000):
    print("before ---------------------")
    for name,param in net.named_parameters():
        print(name," ",param.data)

    output = net(input)
    loss = loss_fn(output,target)
    optimizer.zero_grad()    
    loss.backward()
    optimizer.step()

    print("after ---------------------")
    for name,param in net.named_parameters():
        print(name," ",param.data)
    time.sleep(0.5)
</code></pre>
<p>I got nan values  after loss backward()</p>
<pre><code class="lang-auto">before ---------------------
w   tensor([[[[ 1.0108,  0.8858, -0.5732],
          [-1.1128, -0.8122, -0.7874],
          [ 0.1894,  0.4039,  0.6233]]]])
myParam   tensor([0.5857])
after ---------------------
w   tensor([[[[nan, nan, nan],
          [nan, nan, nan],
          [nan, nan, nan]]]])
</code></pre>

G.M: <p>Can you print the value from <code>self.w**self.myParam</code>? I think this line produced Nan because -0.8122^0.5857 is undefined(for other negative values too). Then, every operation involving Nan result in Nan.</p>

Suyoung_Park: <p>oh you were right!!! That line produced Nan values. Now I see why did you ask me to print out.</p>
<pre><code class="lang-auto">------------self.w--------

 Parameter containing:
tensor([[[[-0.3731,  2.3884,  0.9738],
          [-0.9576,  0.7097, -0.3284],
          [-0.1237, -0.4981, -0.0105]]]], requires_grad=True)

---------self.myParam--------

 Parameter containing:
tensor([0.2613], requires_grad=True)


---self.w**self.myParam---

 tensor([[[[   nan, 1.2555, 0.9931],
          [   nan, 0.9143,    nan],
          [   nan,    nan,    nan]]]], grad_fn=&lt;PowBackward1&gt;)


----------tempVar---------

 tensor([[[[    nan,  1.2555,  0.9931],
          [    nan,  0.9143,     nan],
          [-0.1237,     nan, -0.0105]]]], grad_fn=&lt;SWhereBackward&gt;)

</code></pre>
Using Certain Layers of Mask RCNN/Faster RCNN

uertenli: <p>Hi!</p>
<p>I am trying to use the pretrained Mask RCNN model directly from PyTorch. However, I would like to separate the backbone part from the RPN and the head part, i.e, I want to create two models from the initial model. I have seen many people ask this but, could not find a solution:</p>
<p>If I divide the model using <code>modules = list(model.children())</code> and try to create an <code>nn.Sequential()</code> using this, I get this error: <code>TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not tuple</code>.</p>
<p>Some people also faced this issue, but I couldn’t solve this. Some examples are:</p>
<p><a href="https://stackoverflow.com/questions/59816287/how-to-remove-certain-layers-from-fastern-rcnn-in-pytorch?answertab=oldest#tab-top" rel="nofollow noopener">https://stackoverflow.com/questions/59816287/how-to-remove-certain-layers-from-fastern-rcnn-in-pytorch?answertab=oldest#tab-top</a></p>
<p><a href="https://stackoverflow.com/questions/56455302/use-only-certain-layers-of-pretrained-torchvision-network" rel="nofollow noopener">https://stackoverflow.com/questions/56455302/use-only-certain-layers-of-pretrained-torchvision-network</a></p>
<p>The problem is caused due to the GeneralizedRCNNTransform() returning an <code>torchvision.models.detection.image_list.ImageList</code> object.</p>
<p>Would be grateful for any kind of hints/help.</p>
<p>Thanks in advance.</p>

ptrblck: <p><code>nn.Sequential</code> is a container used for simple feedforward models.<br>
In case of more complicated models, you would have to be careful to not lose any functional API calls in the original <code>forward</code> method.</p>
<p>That being said, I would recommend to either derive your custom class from the base class and manipulate the module as you wish or alternatively create a custom model class and pass in the desired submodules from the pretrained model.</p>
Difference of 0*a + b and b tensor

LeeYongHyeok: <p>Hellow everyone.</p>
<p>Is there any difference between 0*a + b tensor and b tensor in autograd?</p>
<p>In detailed, if tensor <code>y=0*x+b</code> and <code>y=b</code> is same at the back-propagation stage?</p>
<p>Thanks,</p>

JuanFMontesinos: <p>The former one requires more computation as it will compute grad_x *0 but the result itself is the same</p>

LeeYongHyeok: <p>Thank you for reply!</p>
Columnwise/elementwise operations

jacobcvt12: <p>Is it possible to perform a column-wise operation element-wise? For example, if I have the following tensors</p>
<pre><code class="lang-auto">A = torch.ones(3, 2)
B = torch.ones(3)
</code></pre>
<p>I would like to multiply each column (elementwise) of <code>A</code> with with <code>B</code>, which would result in a (3x2) matrix of 2s, or add each column of <code>A</code> with <code>B</code> which would result in a (3x2) matrix of 3s.</p>
<p>I recognize that I can do this procedure with a for loop, but want to know if there’s a way to do it “within” pytorch.</p>

Deeply: <p>You can use  <code>torch.mul()</code><br>
<a href="https://pytorch.org/docs/stable/torch.html#torch.mul" rel="nofollow noopener">Here</a> you can find more info</p>

ajayunagar: <p>I don’t think torch.mul works if two tensors are not broadcastable.</p>
<p>For example,<br>
x = torch.ones(3,2)<br>
y = torch.ones(3)<br>
torch.mul(x,y)</p>
<p>Gives an error:<br>
<em>RuntimeError: The size of tensor a (2) must match the size of tensor b (3) at non-singleton dimension 1</em><br>
I am yet not able to find solution to how to multiply all columns of one tensor with other tensor</p>
<p>Ok found the solution:</p><aside class="quote quote-modified" data-post="1" data-topic="9333">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/blaze/48/1322_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/multiply-1d-tensor-by-2d-tensor/9333">Multiply 1D tensor by 2D tensor</a> 
  </div>
  <blockquote>
    <a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/12b670af5d04f1bbf4747d2e8e55bb360144e826.png" data-download-href="https://discuss.pytorch.org/uploads/default/12b670af5d04f1bbf4747d2e8e55bb360144e826" title="TensorMultiplication.png">[TensorMultiplication]</a> 
I am trying to multiply a 1D tensor by a 2D tensor as shown in the picture above (whereby each element of the first tensor is multiplied by each element in the corresponding row of the second tensor). If I were to run a*b, I would get RuntimeError: inconsistent tensor size. I could of course iterate through each tensor and multiply each of the corresponding elements, but I assume that there exists a ‘better’ way than this?
  </blockquote>
</aside>
Handling unbalanced dataset

gavrilo: <p>Hi,</p>
<p>I’m training SSD with ResNet as a base network, it’s an object detection task. Currently I’m experiencing issue because lots of my input images are actually without any object (just a background or not labeled objets). My dataset contain about 21000 images and only 17% contains object of interest.</p>
<p>I’m facing strong overfitting issue. Do you guys have any idea how to solve it? I’ve tried several approaches, but let’s hear your opinion first <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>Thanks a lot!</p>

ayalaa2: <p>Could you describe your dataset more? I’m going to assume that by “background or not labeled objects”, you mean images that are labeled as “no object”.</p>
<p>Overfitting can be caused by a few issues, I would perhaps suggest balancing your dataset such that the number of images that are background images are balanced with images containing objects. However, as a result you may not have enough data to train on. You could then use a model that was pretrained on a similar task and fine-tune it with your training set.</p>
<p>That would be my immediate thought, but there’s other ways of tackling this issue.</p>

gavrilo: <aside class="quote no-group" data-username="ayalaa2" data-post="2" data-topic="72928">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ayalaa2/48/21195_2.png" class="avatar"> ayalaa2:</div>
<blockquote>
<p>I’m going to assume that by “background or not labeled objects”, you mean images that are labeled as “no object”.</p>
</blockquote>
</aside>
<p>They are labeled, but no objects are on them. My model should output no “boxes” in these cases. For example let’s assume my dataset contains K images and only small percent of them actually have cats on them. The rest of dataset contains random images of other objects, animals etc. And my model should properly detect box where cat is. Is this helpful description? <a class="mention" href="/u/ayalaa2">@ayalaa2</a><br>
Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
Appending tensors

yegane: <p>I want to append a new row to a tensor in each step of a loop, but neither torch.cat nor torch.stack is working for me, because they return an error. It says the tensors must have the same shape.<br>
I can append the tensors to a list, and then stack them, but I wanted to do it in GPU, could you please suggest a way?</p>
<p>e.g. I have a 1x10 tensor, I want to add another 1x10 tensor so I could have 2x10 tensor. It works, but for appending another 1x10 to the 2x10 tensor, so I would have 3x10 tensor.</p>

G.M: <p><code>torch.cat</code> works for me:</p>
<pre><code class="lang-auto">a = tc.empty(1, 10)
for i in range(6):
    b = tc.empty(1, 10)
    a = tc.cat((a, b))
</code></pre>
<p>I suggest appending them to a list and stack them. You can test which way is faster on your GPU.</p>
How "fan_in" and "fan_out" work in "torch.nn.init"?

Nikronic: <p>Hello everyone</p>
<p>I am trying to initialize the layers of my network using <code>kaiming_normal_</code> initializer. This method has an argument called <code>mode</code> with <code>fan_in</code> and <code>fan_out</code> options.</p>
<p>I read the docs and found out it depends on the number of filters or channels. For example, I have a <code>Conv2d</code> layer with size of  <code>[64, 3, 4, 4]</code>. When I calculate the values, I get <code>fan_in=48</code> and <code>fan_out=1024</code> which is a huge difference and result in <code>std = gain/math.sqrt(fan)</code> is like this:</p>
<p><code>fan_in</code> enabled: <code>std=0.2</code><br>
<code>fan_out</code> enabled: <code>std=0.04</code></p>
<p>And <code>std</code> is passed to <code>tensor.normal_</code> method.</p>
<p>The question is I did not find any specific explanation about which <code>mode</code> should be chose in the paper I am trying to implement or other resources.</p>
<p>Best regards</p>

avinash_m: <p>As far as I know, kaiming_normal_ or he_normal is generally initialized using fan_in.</p>

tom: <p>There are two parts:</p>
<ul>
<li>As Avinash points out, the default mode <code>'fan_in'</code> is probably a good choice.</li>
<li>For some intuition of why this is: Each output is a weighted sum of <code>fan_in</code> inputs. For linear, this is one row of the matrix multiplication, for convolutions it is number of in-channels  * kernel size.</li>
</ul>
<p>When <a class="mention" href="/u/ptrblck">@ptrblck</a> and I implemented <a href="https://github.com/lernapparat/lernapparat/blob/master/style_gan/pytorch_style_gan.ipynb" rel="nofollow noopener">StyleGAN</a> for PyTorch (but I don’t think the StyleGAN authors necessarily invented it), I’ve come across the idea of not having the multiplier used during init, but applying them to the weight before using it. This has the effect of using the scaling for both initial values and gradient updates. (Even if it is not that efficient without a hand-made convolution kernel.)</p>
<p>Best regards</p>
<p>Thomas</p>

X-funbean: <p>Why dose resnet in torchvision apply kaiming_normal_ with mode ‘fan_out’? Is there a specific reason to do so?<br>
<a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L156" rel="nofollow noopener">https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L156</a></p>