Difficulties building a FCN transfer learning model for binary segmentation

username: Patrice
cooked: <p>I just started learning Pytorch and I do not have good programming skills. I am trying to perform a segmentation task and given the limited amount of training samples, I have opted for transfer learning approach. I have managed to run the following model with my own data and the result is very bad.</p>
<pre><code class="lang-auto">class fcn(nn.Module):
    def __init__(self, num_classes):
        super(fcn, self).__init__()
        self.stage1 = nn.Sequential(*list(pretrained_net.children())[:-4])
        # change input channels to 9
        self.stage1[0] = nn.Conv2d(9, 64, kernel_size=7, stride=2, padding=3, bias=False)
        self.stage2 = list(pretrained_net.children())[-4]
        self.stage3 = list(pretrained_net.children())[-3]
        self.scores1 = nn.Conv2d(512, num_classes, 1)
        self.scores2 = nn.Conv2d(256, num_classes, 1)
        self.scores3 = nn.Conv2d(128, num_classes, 1)
        self.upsample_8x = nn.ConvTranspose2d(num_classes, num_classes, 16, 8, 4, bias=False)
        self.upsample_4x = nn.ConvTranspose2d(num_classes, num_classes, 4, 2, 1, bias=False)
        self.upsample_2x = nn.ConvTranspose2d(num_classes, num_classes, 4, 2, 1, bias=False)

    def forward(self, x):
        x = self.stage1(x)
        s1 = x  # 1/8
        x = self.stage2(x)
        s2 = x  # 1/16
        x = self.stage3(x)
        s3 = x  # 1/32
        s3 = self.scores1(s3)
        s3 = self.upsample_2x(s3)
        s2 = self.scores2(s2)
        s2 = s2 + s3
        s1 = self.scores3(s1)
        s2 = self.upsample_4x(s2)
        s = s1 + s2
        s = self.upsample_8x(s2)
        return s


num_classes = 2 #len(classes)

pretrained_net = models.resnet18(pretrained=True)

model = fcn(num_classes)
</code></pre>
<p>With this same configuration, the model could only run with Resnet18 and Resnet32. When I tried other network structures such as Resnet50, VGG, Mobilenet. I got the following error</p>
<pre><code class="lang-auto"> self.stage1[0] = nn.Conv2d(9, 64, kernel_size=7, stride=2, padding=3, bias=False)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\container.py", line 71, in __setitem__
    key = self._get_item_by_idx(self._modules.keys(), idx)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\container.py", line 60, in _get_item_by_idx
    raise IndexError('index {} is out of range'.format(idx))
IndexError: index 0 is out of range
</code></pre>
<p>Please, how can I solve this problem? How can I try a transfer learning framework exploring different network structures for a segmentation task? Any suggestions and comments would be highly appreciated.</p>

username: ptrblck
cooked: <p>I don’t get the error message using <code>resnet50</code> as the <code>pretrained_net</code>.<br>
However, you are currently not registering <code>stage2</code> and <code>stage3</code> properly, since you are using a Python list. Use <code>nn.ModuleList</code> instead. Otherwise, <code>model.parameters()</code> won’t return these parameters and <code>model.to()</code> won’t push the parameters to the specified device.</p>

username: Patrice
cooked: <p>Thank you for your reply sir. I have replaced  ‘list’ with <code>nn.ModuleList</code> as you suggested and after adjusting the number of neurons in the convolutional layers, Resnet50 could run. However, VGG and MobileNet still gave the same error. I guess this might be due to some differences in network architectures. Please, could you explain to me how to fix this? I would also like to get better understanding of the design of a transfer learning based FCN model for segmentation problems.</p>
<p>Really appreciate your time, patience and guidance</p>

username: ptrblck
cooked: <p>The error are most likely created by trying to create <code>stage1</code> using <code>nn.Sequential</code>:</p>
<pre><code class="lang-python">stage1 = nn.Sequential(*list(pretrained_net.children())[:-4])
</code></pre>
<p>Since each architecture is different, this approach, and the indexing in particular, might not be suitable.<br>
Could you check, if the stages are really containing, what you expect them to?</p>

username: Patrice
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="72364">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>stage1 = nn.Sequential(*list(pretrained_net.children())[:-4])</p>
</blockquote>
</aside>
<p>I think that this line means all but the last 4 layers of the pretrained model. Please, correct me if I am wrong.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="72364">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Could you check, if the stages are really containing, what you expect them to?</p>
</blockquote>
</aside>
<p>I guess, I can only index the fully connected layers of a selected network architecture. So, the indexing must be related to the number of fully connected layers in the network structure given that building FCN of an CNN simply consists of replacing the FC layers with convolutional layers.  Sorry if it might be a little weird. But ,I am just trying to get it right.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Patrice" data-post="5" data-topic="72364">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/e274bd/48.png" class="avatar"> Patrice:</div>
<blockquote>
<p>I think that this line means all but the last 4 layers of the pretrained model. Please, correct me if I am wrong.</p>
</blockquote>
</aside>
<p>Yes, that;s correct. While this might work for resnet50, it might not for other architectures.</p>
<aside class="quote no-group" data-username="Patrice" data-post="5" data-topic="72364">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/e274bd/48.png" class="avatar"> Patrice:</div>
<blockquote>
<p>I guess, I can only index the fully connected layers of a selected network architecture. So, the indexing must be related to the number of fully connected layers in the network structure given that building FCN of an CNN simply consists of replacing the FC layers with convolutional layers. Sorry if it might be a little weird. But ,I am just trying to get it rig</p>
</blockquote>
</aside>
<p>That might be the simplest method. Classification models usually have a classification layer created as <code>model.fc</code> or <code>model.classifier</code>, which you could replace with your custom convs.</p>

username: Patrice
cooked: <p>Thank you for the explanation.</p>