How to modify Deeplabv3 and FCN models for grayscale images

Nishanth_Sasankan: <p>Hi All,</p>
<p>How can I modify the deeplabv3_resnet101 and fcn_resnet101 models available from torchvision segmentation models to accept input images with only 1 color channel?</p>
<p>I have seen some example of how I can modify resnet, but I am not sure  how to do it for these</p>
<p>Thanks<br>
Nishanth</p>

ptrblck: <p>You could replace the first conv layer in a similar manner as done with resnet:</p>
<pre><code class="lang-python">model = torchvision.models.segmentation.deeplabv3_resnet101()
model.backbone.conv1 = nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3, bias=False)
x = torch.randn(2, 1, 224, 224)
output = model(x)
</code></pre>

Dorra: <p>hello</p>
<p>I faced the same issue, I have tried to fine-tune deeplabv3_resnet50/resnet101 and fcn_resnet50/resnet101 on grayscale images(one channel as input). I tried your solution mentioned above but it’s not working, this is the code that I m using<br>
class FCN_resnet50(nn.Module):<br>
def <strong>init</strong>(self, num_classes=2, pretrained=False):<br>
super(FCN_resnet50, self).<strong>init</strong>()<br>
model=fcn_resnet50(pretrained)<br>
# model.eval()<br>
# model=timm.create_model(‘fcn_resnet50’, pretrained, in_chans=1)</p>
<pre><code>    # take pretrained resnet, except AvgPool and FC
    # self.conv_weight = model.conv1[0].weight
    #     # nn.Conv2d(1, self.inplanes, kernel_size=3, padding=1,
    #     #                    bias=False)
    # model.conv1[0].in_channels=1
    # model.conv1[0].weight=torch.nn.parameter(self.conv_weight.sum(dim=1, keepdim=True))
    # weight1 = model.features.conv1.weight.clone()
    # new_first_layer = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3),
    #                             bias=False).requires_grad_()
    # new_first_layer.weight[:, :3, :, :].data[...] = Variable(weight1, requires_grad=True)
    # model.features.conv1 = new_first_layer
    model.backbone.conv1 = nn.Conv2d(1, 64, kernel_size=7, stride=2, padding=3, bias=False)

    self.conv1=model.backbone.conv1
    self.bn1 = model.bn1
    self.relu = model.relu
    self.maxpool = model.maxpool
    self.layer1 = model.layer1
    self.layer2 = model.layer2
    self.layer3 = model.layer3
    self.layer4 = model.layer4
    self.last_conv = nn.Conv2d(2048, num_classes, kernel_size=1)
</code></pre>
<p>As you see I tried multiple instructions but no one worked for me.</p>
<p>I will appreciate your help.</p>
<p>many thanks.</p>

ptrblck: <p>Your code is unfortunately not formatted correctly and also not executable.<br>
What kind of issue are you seeing using my code snippet?</p>

Dorra: <p>RuntimeError: Given groups=1, weight of size [64, 3, 7, 7], expected input[16, 1, 384, 384] to have 3 channels, but got 1 channels instead</p>

ptrblck: <p>I just executed my code snippet again and don’t see the error. Are you sure you are running <a href="https://discuss.pytorch.org/t/how-to-modify-deeplabv3-and-fcn-models-for-grayscale-images/52688/2">this code</a>?</p>

Dorra: <p>yes I m sure that using this model, my aim is to modify deeplabv3_resnet50/resnet101 and fcn_resnet50/resnet101 to segment medical imaging that is stored in 2d grayscale images<br>
then I load the model through pytorch lightning module.</p>

ptrblck: <p>I meant to ask if you were executing my posted code snippet directly as I do not see the mentioned error. If you manipulate my code or are using it in another way, your change might cause the issue in which case a minimal and executable code snippet would be needed to help debugging it.</p>

Dorra: <p>the problem that i m generating 3 outputs from forward method so I m trying to modify this method too<br>
this if the full scrip<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/5/95fca90d78e72f1b4444597d7c44eadf351a6e14.png" data-download-href="https://discuss.pytorch.org/uploads/default/95fca90d78e72f1b4444597d7c44eadf351a6e14" title="Screenshot from 2023-01-13 20-19-38"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/95fca90d78e72f1b4444597d7c44eadf351a6e14_2_496x500.png" alt="Screenshot from 2023-01-13 20-19-38" data-base62-sha1="loQsR5WqmEMocJMU97QtKgIcSws" width="496" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/95fca90d78e72f1b4444597d7c44eadf351a6e14_2_496x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/95fca90d78e72f1b4444597d7c44eadf351a6e14_2_744x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/5/95fca90d78e72f1b4444597d7c44eadf351a6e14.png 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-01-13 20-19-38</span><span class="informations">770×775 62.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>