Break resnet into two parts

mkt1412: <p>Hi, all:</p>
<p>I try to manipulate some intermediate features of resnet, so i have to break pre-trained model into two parts.<br>
I feed original image into first part and then feed the output of first part directly into second part. But i meet size mismatch in last linear layer.</p>
<p>“RuntimeError: size mismatch, m1: [512 x 1], m2: [512 x 1000]”</p>
<pre><code class="lang-auto">modules = list(resnet18.children())[:3]
resnet_1st = nn.Sequential(*modules)
for p in resnet18.parameters():
    p.requires_grad = False

modules = list(resnet18.children())[3:]
resnet_2nd = nn.Sequential(*modules)
for p in resnet18.parameters():
    p.requires_grad = False
#print(resnet_2nd)

out_1st = resnet_1st(image)
print(out_1st.shape)
out_2nd = resnet_2nd(out_1st)
print(out_2nd.shape)

</code></pre>
<p>Any one know how to solve this? Thanks in advance!</p>

ptrblck: <p>The error is thrown, since you are wrapping all modules in an <code>nn.Sequential</code> module, which is missing the flatten operation defined in resnet’s <code>forward</code>.<br>
You could define a custom <code>Flatten</code> module and add it right before the last linear layer:</p>
<pre><code class="lang-python">class Flatten(nn.Module):
    def __init__(self):
        super(Flatten, self).__init__()
        
    def forward(self, x):
        x = x.view(x.size(0), -1)
        return x

modules = list(model.children())[:3]
resnet_1st = nn.Sequential(*modules)

modules = list(model.children())[3:-1]
resnet_2nd = nn.Sequential(*[*modules, Flatten(), list(model.children())[-1]])

x = torch.randn(1, 3, 224, 224)
out_1st = resnet_1st(x)
print(out_1st.shape)
out_2nd = resnet_2nd(out_1st)
print(out_2nd.shape)
</code></pre>

mkt1412: <p>Thank you so much!</p>
<p>唐潮</p>
<p>CHAO TANG</p>

cealia: <p>When splitting a predefined nn.Sequential inorder to get intermediate layer output in forward, can we split it directly like using model.submodel_name[:n] (which I found is still a nn.Sequential)  instead of extracting layers from model.children and wrap them again in a nn.Sequential? Is there any issues to concern about? Thanks.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/d/2d7e1e2fb93961a35647b0f146e15f8b110da4a9.png" data-download-href="https://discuss.pytorch.org/uploads/default/2d7e1e2fb93961a35647b0f146e15f8b110da4a9" title="螢幕快照 2020-11-09 下午11.29.46"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d7e1e2fb93961a35647b0f146e15f8b110da4a9_2_690x145.png" alt="螢幕快照 2020-11-09 下午11.29.46" data-base62-sha1="6urGWc3XAFXAcfshwadpYWeTHrb" width="690" height="145" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d7e1e2fb93961a35647b0f146e15f8b110da4a9_2_690x145.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d7e1e2fb93961a35647b0f146e15f8b110da4a9_2_1035x217.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/d/2d7e1e2fb93961a35647b0f146e15f8b110da4a9_2_1380x290.png 2x" data-dominant-color="ECECEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">螢幕快照 2020-11-09 下午11.29.46</span><span class="informations">1674×352 64 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Your approach should work:</p>
<pre><code class="lang-python">model = nn.Sequential(
    nn.Conv2d(3, 6, 3, 1, 1),
    nn.ReLU(),
    nn.MaxPool2d(2),
    nn.Conv2d(6, 12, 3, 1, 1),
    nn.ReLU(),
    nn.Flatten(),
    nn.Linear(12*12*12, 10),
)

x = torch.randn(10, 3, 24, 24)
out_ref = model(x)

out1 = model[:5](x)
out2 = model[5:](out1)

print((out_ref - out2).abs().max())
&gt; tensor(0., grad_fn=&lt;MaxBackward1&gt;)
</code></pre>

STK: <p>I tried to implement this code and working fine. But if I try to modify last layer then getting error. I am new to pytorch so not able to figure out what’s wrong. Only understood that there is some dimension mismatch due to last linear layer. Please help me to resolve the issue. The code I am running is as follows:</p>
<p>from torchvision import models</p>
<p>import torch</p>
<p>import torch.nn as nn</p>
<p>class Flatten(nn.Module):</p>
<pre><code>def __init__(self):

    super(Flatten, self).__init__()

    

def forward(self, x):

    x = x.view(x.size(0), -1)

    return x
</code></pre>
<p>model = models.resnet152(pretrained=True)</p>
<p>modules = list(model.children())[:6]</p>
<p>resnet_1st = nn.Sequential(*modules)</p>
<p>modules = list(model.children())[6:-1]</p>
<p>resnet_2nd = nn.Sequential(*[*modules, Flatten(), list(model.children())[-1]])</p>
<p>model.fc = nn.Linear(model.fc.in_features, 512)</p>
<p>x = torch.randn(1, 3, 224, 224)</p>
<p>out_1st = resnet_1st(x)</p>
<p>print(out_1st.shape)</p>
<p>out_2nd = resnet_2nd(out_1st)</p>
<p>print(out_2nd.shape)</p>
<p>for param in model.parameters():</p>
<pre><code>param.requires_grad = False
</code></pre>
<p>out = model.fc(out_2nd)</p>
<p>print(out.shape)</p>
<h2>
<a name="error-i-am-getting-torchsize1-512-28-28-torchsize1-1000-1" class="anchor" href="#error-i-am-getting-torchsize1-512-28-28-torchsize1-1000-1"></a>Error I am getting:<br>
torch.Size([1, 512, 28, 28])<br>
torch.Size([1, 1000])</h2>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
26 for param in model.parameters():<br>
27     param.requires_grad = False<br>
—&gt; 28 out = model.fc(out_2nd)<br>
29 print(out.shape)</p>
<p>2 frames<br>
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)<br>
1846     if has_torch_function_variadic(input, weight, bias):<br>
1847         return handle_torch_function(linear, (input, weight, bias), input, weight, bias=bias)<br>
 → 1848     return torch._C._nn.linear(input, weight, bias)<br>
1849<br>
1850</p>
<p>RuntimeError: mat1 and mat2 shapes cannot be multiplied (1x1000 and 2048x512)</p>

ptrblck: <p><code>resnet_2nd</code> already contains the last linear layer (<code>model.fc</code>) as you are explicitly adding it in:</p>
<pre><code class="lang-python">resnet_2nd = nn.Sequential(*[*modules, Flatten(), list(model.children())[-1]])
</code></pre>
<p>after the custom <code>Flatten()</code> module.<br>
Later you are replacing the <code>model.fc</code> layer with a new <code>nn.Linear</code> layer, which won’t change <code>resnet_2nd</code> anymore.<br>
<code>out_2nd</code> is thus the output of the original last linear layer and has a shape of <code>[batch_size, 1000]</code>.<br>
In this line of code:</p>
<pre><code class="lang-python">out = model.fc(out_2nd)
</code></pre>
<p>you are then trying to pass this output again to the newly initialized <code>model.fc</code> layer which will create the shape mismatch.</p>

STK: <p>Thank you for the information.</p>
<p>Actually I want to split the resnet as above. Also I want to reduce the output dimension to 512 in the FC layer. How to do that? I tried a lot but could not. Please help me.</p>

ptrblck: <aside class="quote no-group" data-username="STK" data-post="8" data-topic="39315">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stk/48/45130_2.png" class="avatar"> STK:</div>
<blockquote>
<p>Actually I want to split the resnet as above.</p>
</blockquote>
</aside>
<p>It wouldn’t make sense to reuse the last linear layer.</p>
<aside class="quote no-group" data-username="STK" data-post="8" data-topic="39315">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/stk/48/45130_2.png" class="avatar"> STK:</div>
<blockquote>
<p>Also I want to reduce the output dimension to 512 in the FC layer. How to do that? I tried a lot but could not.</p>
</blockquote>
</aside>
<p>Unsure what you’ve tried, but replacing the layer before creating the <code>nn.Sequential</code> container should work:</p>
<pre data-code-wrap="python"><code class="lang-python">modules = list(model.children())[6:-1]
model.fc = nn.Linear(model.fc.in_features, 512)
resnet_2nd = nn.Sequential(*[*modules, Flatten(), list(model.children())[-1]])
</code></pre>

STK: <p>Hi, I have a situation where i have two dataloaders. i.e. Source1_loader and Source2_loader. I want to train batch of data from both of these dataloaders simultaneously on resnet_1st (here I want to use resnet_1st to learn data from both the datasets). I tried to implement it but not getting proper results.<br>
I am feeling that network is not learning from second source. Please help me to solve the issue. My code is</p>
<p>model = models.resnet152(pretrained=True)<br>
modules = list(model.children())[:7]<br>
self.resnet_1st = nn.Sequential(*modules)</p>
<p>modules = list(model.children())[7:-1]<br>
self.net1 = nn.Sequential(*[*modules, Flatten(), list(model.children())[-1]])</p>
<p>modules = list(model.children())[7:-1]<br>
self.net2 = nn.Sequential(*[*modules, Flatten(), list(model.children())[-1]])<br>
for param in model.parameters():<br>
param.requires_grad = False</p>
<p>In forward part:<br>
data_src1 = self.resnet_1st (data_src1)                # Extract Features of Source1<br>
data_src2 = self.resnet_1st (data_src2)                # Extract Features of Source2</p>
<p>data_src1 = self.net1(data_src1)<br>
data_src2 = self.net2(data_src2)</p>

ptrblck: <p>You are freezing all models, since they are sharing parameters:</p>
<pre><code class="lang-python">for param in model.parameters():
    param.requires_grad = False

for name, param in net1.named_parameters():
    print(name, param.requires_grad)
    
for name, param in net2.named_parameters():
    print(name, param.requires_grad)
</code></pre>
<p>will return <code>False</code> for <code>net1</code> and <code>net2</code>.<br>
Use <code>copy.deepcopy</code> if you want to create copies of the models.</p>

BasedLukas: <p>I am trying to do something similar, with the intention on using the features for an OCR model. The model needs to intelligently decide what text is relevant. So I was thinking of using “high level” features as well as “low level features”. The intention is to use the resnet as an encoder, and use a transformer based model as a decoder, with cross attn on the resnet features. I am using the following code to do this:</p>
<pre><code class="lang-auto">class CustomResNet(nn.Module):
    def __init__(self, out_dim):
        super(CustomResNet, self).__init__()
        resnet = models.resnet152(weights=models.ResNet152_Weights.DEFAULT)
        
        self.first = nn.Sequential(*list(resnet.children())[:6])
        self.second = nn.Sequential(*list(resnet.children())[6:-2])
        self.out_proj = nn.Linear(501760, out_dim)
        
    def forward(self, x):
        # Forward pass to the point of interest
        intermediate = self.first(x)
        x = self.second(intermediate)

        #flatten and concat
        x = torch.flatten(x, 1)
        intermediate = torch.flatten(intermediate, 1)
        concatenated_features = torch.cat((x, intermediate), dim=1)

        out = self.out_proj(concatenated_features)
        return out
</code></pre>
<p>Does my intuition and implementation here make sense? Is there a known/tested method to reducing the conditionality of the features (from 501760 in my instance) to something more reasonable?</p>