Resnet last layer modification

Paulo_Mann: <p>Hello guys, I’m trying to add a dropout layer before the FC layer in the “bottom” of my resnet. So, in order to do that, I remove the original FC layer from the resnet18 with the following code:</p>
<pre><code class="lang-auto">    resnetk = models.resnet18(pretrained=True)
    num_ftrs = resnetk.fc.in_features
    resnetk = torch.nn.Sequential(*list(resnetk.children())[:-1])
</code></pre>
<p>Then, I add the dropout and the FC layer using the num_ftrs I obtained from the previous (original) FC layer of my resnet18:</p>
<pre><code class="lang-auto">    resnetk.add_module("dropout", nn.Dropout(p=0.5))
    resnetk.add_module("fc", nn.Linear(num_ftrs, n_classes))

</code></pre>
<p>But I receive the following error : RuntimeError: size mismatch, m1: [8192 x 1], m2: [512 x 2] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:266</p>
<p>I’m also confused where the softmax gets in, after the linear layer, since in Keras we need to specify the activation function as softmax.</p>

ptrblck: <p>Currently you are rewrapping your pretrained resnet into a new <code>nn.Sequential</code> module, which will lose the <code>forward</code> definition. As you can see in <a href="https://github.com/pytorch/vision/blob/21153802a3086558e9385788956b0f2808b50e51/torchvision/models/resnet.py#L161" rel="nofollow noopener">this line of code</a> in the original resnet implementation, the activation <code>x</code> will be flattened before being passed to the last linear layer. Since this is missing now, you’ll get the size mismatch error.</p>
<p>You could just manipulate the <code>model.fc</code> attribute and add your dropout and linear layer:</p>
<pre><code class="lang-python">model = models.resnet18(pretrained=True)
model.fc = nn.Sequential(
    nn.Dropout(0.5),
    nn.Linear(num_ftrs, 10)
)
</code></pre>
<p>This approach will keep your <code>forward</code> method.</p>
<p>It depends on the criterion you are using if you should add a final non-linearity in your model.<br>
If you are dealing with a classification use case and would like to use <code>nn.CrossEntropyLoss</code>, you should pass the raw logits to the loss function (i.e. no final non-linearity), since <code>nn.LogSoftmax</code> and <code>nn.NLLLoss</code> will be called internally.<br>
However, if you would like to use <code>nn.NLLLoss</code>, you should add the <code>nn.LogSoftmax</code> manually.</p>

Paulo_Mann: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, your comments and observations helped me to understand my problem. And I appreciate that you pointed the line of code in the original resnet implementation, it helped even more!</p>

ronitganguly: <p>Hi,<br>
Isn’t it like adding dropout to the last layer? Why do we want to drop out some outputs?</p>
<p>Regards,</p>

ptrblck: <p>No, as the dropout layer is added before the output layer not after.</p>

ronitganguly: <p>I am very new in this field. Would you agree with me if I say: since it’s sequentially built, whatever written first is applied before. And functional.dropout of a layer is applied after because in the forward method, we are calling the layers one after another, so dropout is required after the layer has been activated?</p>

ptrblck: <p>The execution order of each module in an <code>nn.Sequential</code> container is the same order their <code>forward</code> methods will be called.<br>
Here is a small example, which shows that the first case yields a dense output, while the second one zeroes out some output units:</p>
<pre><code class="lang-python"># Case 1
model = nn.Sequential(
    nn.Dropout(0.5),
    nn.Linear(10, 10)
)

x = torch.randn(1, 10)
out = model(x)
print(out)

# Case 2
model = nn.Sequential(
    nn.Linear(10, 10),
    nn.Dropout(0.5)
)

x = torch.randn(1, 10)
out = model(x)
print(out)
</code></pre>

ronitganguly: <aside class="quote no-group" data-username="ptrblck" data-post="7" data-topic="33530">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>model = nn.Sequential( nn.Dropout(0.5), nn.Linear(10, 10) ) x = torch.randn(1, 10) out = model(x) print(out) # Case 2 model = nn.Sequential( nn.Linear(10, 10), nn.Dropout(0.5) ) x = torch.randn(1, 10) out = model(x)</p>
</blockquote>
</aside>
<p>Thank you so much for this example.<br>
Could you tell me what we are dropping in Resnet before the last layer because there is no linear layer before the last layer but a convolutional layer?</p>
<p>Regards</p>

ptrblck: <p>In this example, you would drop some of the features, which are fed into the linear layer.<br>
I just provided the code snippet for <a class="mention" href="/u/paulo_mann">@Paulo_Mann</a>, so he might know better what his use case is. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

ronitganguly: <p>I always thought that Dropout is meant to drop some neurons in a Linear layer to reduce the outgoing features. I did not know that it is used to drop in-coming features as well.</p>
<p>Thank you very much for this information. Also, if you have a source handy where I can read more about dropouts’ this behavior, could you please provide <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>BR,</p>

Ripley: <p>Can I ask, when I tried to add another additional linear layer, but it came out with</p>
<p>RuntimeError: Error(s) in loading statam e_dict for ResNet: size mismatch for fc.2.weight: copying a param with shape torch.Size([20, 100]) from checkpoint, the shape in current model is torch.Size([60, 100]). size mismatch for fc.2.bias: copying a parwith shape torch.Size([20]) from checkpoint, the shape in current model is torch.Size([60]). size mismatch for fc.5.weight: copying a param with shape torch.Size([4, 20]) from checkpoint, the shape in current model is torch.Size([20]). size mismatch for fc.5.bias: copying a param with shape torch.Size([4]) from checkpoint, the shape in current model is torch.Size([20]).</p>
<p>I’ve already state strict = False, when i load my model. What else could I be doing wrong?</p>
<h1>def initModel4():</h1>
<h1>resnetbase = models.resnet34(pretrained=True).to(device)</h1>
<h1>for param in resnetbase.parameters(): <span class="hashtag">#Freeze</span> all the layers and train only the last layer</h1>
<h1>param.requires_grad = False</h1>
<h1>#   add in dropout layer</h1>
<h1>fc_layers = nn.Sequential(</h1>
<h1>nn.Linear(resnetbase.fc.in_features,100),</h1>
<h1>nn.Hardswish(),</h1>
<h1>nn.Linear(100, 20),</h1>
<h1>nn.BatchNorm1d(20),</h1>
<h1>nn.Dropout(0.12),</h1>
<h1>nn.Linear(20, 10),</h1>
<h1>nn.Linear(10, 4),</h1>
<h1>).to(device)</h1>
<h1>#<span class="hashtag">#weights</span> for background imgs, tanks, floating head tanks, tank clusters</h1>
<h1>class_weights = torch.FloatTensor([0.1,0.5,0.3,2.0]).to(device)</h1>
<h1>loss_function = nn.CrossEntropyLoss(torch.FloatTensor([0.2,0.6,0.4,2.0])).to(device)</h1>
<h1>optimizer = torch.optim.Adam(resnetbase.parameters(), lr=lr)</h1>
<h1>scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1)</h1>
<h1>return</h1>

ptrblck: <p>If you want to change some layers in a pretrained model, make sure to load the <code>state_dict</code> containing the parameters and buffers in the expected shapes before appyling any manipulations, such as assigning new layers to attributes.<br>
<code>strict=False</code> wouldn’t work in this case, as there are no key mismatches, but the shapes are wrong.</p>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Ripley: <p>Thank you once again! Apologies for the late reply as I was working on other work for awhile.<br>
And opps! Thanks for the tip! Will take note of it next time!<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

saba: <p>Hi Ptrblck,</p>
<p>I am trying to add extra layer like Resnet, to the input layer and then pass it to the next layer. (not exact as Resnet, by adding conv3d  (self.l33 and self.l44)  between my generator layers, I want to add more features to the generator)</p>
<p>Should I use "Conv3d or Conv1d " in the last layer (self.l6). I think Cond3d is correct. Can I write the last layer with Conv1d?</p>
<pre><code class="lang-auto">ngpu=1
nz=11
ngf=8

class Generator(nn.Module):
    def __init__(self,ngpu,nz,ngf):
        super(Generator, self).__init__()
        self.ngpu=ngpu
        self.nz=nz
        self.ngf=ngf
   
        ## ---1x11x1x1x1
        self.l1= nn.Sequential(
            nn.ConvTranspose3d(self.nz+1, self.ngf * 6, 3, 1, 0, bias=True),
            nn.BatchNorm3d(self.ngf * 6),
            nn.ReLU(),)
        ##---48x3x3x3
        self.l2=nn.Sequential(nn.ConvTranspose3d(self.ngf * 6, self.ngf * 4, 3, 2, 0, bias=True),
            nn.BatchNorm3d(self.ngf * 4),
            nn.ReLU(),)
        ## ---32x7x7x7
        self.l3=nn.Sequential(nn.ConvTranspose3d( self.ngf * 4, self.ngf * 2, 3, 1, 0, bias=True),
            nn.BatchNorm3d(self.ngf * 2),
             nn.ReLU(),)
        ## ----16x9x9x9
        self.l4=nn.Sequential(nn.ConvTranspose3d( self.ngf*2, self.ngf*2, 3, 1, 0, bias=True),nn.BatchNorm3d(self.ngf * 2),
             nn.ReLU(),)
        ## ----16x9x9x9
        self.l44=nn.Sequential(nn.Conv3d(ngf * 2, self.ngf * 2, 3, 1, 1, bias=True),nn.BatchNorm3d(ngf * 2),nn.ReLU())
        ## ---16x11x11x11
        self.l5=nn.Sequential(nn.ConvTranspose3d( self.ngf*2, self.ngf*2, 3, 1, 0, bias=True),nn.BatchNorm3d(self.ngf * 2),
             nn.ReLU(),)
        ## ---16x11x11x11
        self.l55=nn.Sequential(nn.Conv3d(ngf * 2, self.ngf*2, 3, 1, 1, bias=True),nn.BatchNorm3d(ngf * 2),nn.ReLU())
        ## ---1x11x11x11
        self.l6=nn.Sequential(nn.Conv3d( self.ngf*2, 1, 3, 1, 0, bias=True),nn.Sigmoid())

    def forward(self, input,Labels):

        Labels=Labels.unsqueeze(1).unsqueeze(2).unsqueeze(3).unsqueeze(4)
        InputCat=torch.cat((Labels,input),1)
        out1=self.l1(InputCat)
        out2=self.l2(out1)
        out3=self.l3(out2)
        out4=self.l4(out3)    
        out5=self.l44(out4)
        ## ---- add layers---
        out6=out5+out4
        out7=self.l5(out6)
        out8=self.l55(out7)
        ## ---- add layers---
        out9=out8+out7
        out_total=self.l6(out9)
  
        return out_total

batchsize=10
Noise=torch.randn(batchsize,nz,1,1,1)
Condition=torch.ones(size=(batchsize,))

Gen=Generator(ngpu,nz,ngf)
Out=Gen(Noise,Condition)
</code></pre>

ptrblck: <p>The difference between <code>nn.Conv1d</code> and <code>nn.Conv3d</code> is the expected shape of the input (and thus also the kernel shape etc.).<br>
<code>nn.Conv1d</code> would be used for “temporal” signals in the shape <code>[batch_size, channels, seq_len]</code>, while <code>nn.Conv3d</code> is used for “volumetric” signals in the shape <code>[batch_size, channels, depth, height, width]</code>.<br>
Based on the comments from your code you are starting with a 5D tensor, which suddenly gets reduced to a 4D one, so I would assume that either the comments are wrong or the model is not working at the moment.</p>

saba: <p>you mean that you get error from the code? would you please tell me what do you mean by it is not working?<br>
I apply it and did not get any error. before layer6 (self.l6)  bachsize=10, channel=16, depth=11, height=11, width=11, after that I get (10,1,11,11,11) as output.<br>
My aim is to generate 3d patches as 10x1x11x11x11<br>
input is a noise 10x11x1x1x1 which will be concatenated with the Labels and will be 10x12x1x1x1 as the first layer input (1x11x1x1x1 is wrong)</p>