How to change UNET backbone to mobilenetv3_large_100?

nitay_shlump: <p>Below is my code for a UNET model. How can I change its backbone to the CNN model mobilenetv3_large_100? Also, the images I wish to train it on are grayscale.</p>
<pre><code class="lang-auto">class double_conv(nn.Module):
    def __init__(self, in_ch, out_ch):
        super(double_conv, self).__init__()
        self.conv = nn.Sequential(
            nn.Conv2d(in_ch, out_ch, 3, padding=1),
            nn.BatchNorm2d(out_ch),
            nn.ReLU(inplace=True),
            nn.Conv2d(out_ch, out_ch, 3, padding=1),
            nn.BatchNorm2d(out_ch),
            nn.ReLU(inplace=True)
        )

    def forward(self, x):
        x = self.conv(x)
        return x


class inconv(nn.Module):
    def __init__(self, in_ch, out_ch):
        super(inconv, self).__init__()
        self.conv = double_conv(in_ch, out_ch)

    def forward(self, x):
        x = self.conv(x)
        return x


class down(nn.Module):
    def __init__(self, in_ch, out_ch):
        super(down, self).__init__()
        self.mpconv = nn.Sequential(
            nn.MaxPool2d(2),
            double_conv(in_ch, out_ch)
        )

    def forward(self, x):
        x = self.mpconv(x)
        return x


class up(nn.Module):
    def __init__(self, in_ch, out_ch, bilinear=True):
        super(up, self).__init__()


        if bilinear:
            self.up = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)
        else:
            self.up = nn.ConvTranspose2d(in_ch//2, in_ch//2, 2, stride=2)

        self.conv = double_conv(in_ch, out_ch)

    def forward(self, x1, x2):
        x1 = self.up(x1)

        # input is CHW
        diffY = x2.size()[2] - x1.size()[2]
        diffX = x2.size()[3] - x1.size()[3]

        x1 = F.pad(x1, (diffX // 2, diffX - diffX//2,
                        diffY // 2, diffY - diffY//2))



        x = torch.cat([x2, x1], dim=1)
        x = self.conv(x)
        return x


class outconv(nn.Module):
    def __init__(self, in_ch, out_ch):
        super(outconv, self).__init__()
        self.conv = nn.Conv2d(in_ch, out_ch, 1)

    def forward(self, x):
        x = self.conv(x)
        return x

class UNET(nn.Module):
    def __init__(self, n_channels, n_classes):
        super(UNET, self).__init__()
        self.inc = inconv(n_channels, 64)
        self.down1 = down(64, 128)
        self.down2 = down(128, 256)
        self.down3 = down(256, 512)
        self.down4 = down(512, 512)
        self.up1 = up(1024, 256)
        self.up2 = up(512, 128)
        self.up3 = up(256, 64)
        self.up4 = up(128, 64)
        self.outc = outconv(64, n_classes)

    def forward(self, x):
        x1 = self.inc(x)
        x2 = self.down1(x1)
        x3 = self.down2(x2)
        x4 = self.down3(x3)
        x5 = self.down4(x4)
        x = self.up1(x5, x4)
        x = self.up2(x, x3)
        x = self.up3(x, x2)
        x = self.up4(x, x1)
        x = self.outc(x)
        return torch.sigmoid(x)
</code></pre>

KFrank: <p>Hi Nitay!</p>
<aside class="quote no-group" data-username="nitay_shlump" data-post="1" data-topic="205336" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nitay_shlump/48/67956_2.png" class="avatar"> nitay_shlump:</div>
<blockquote>
<p>How can I change its backbone to the CNN model mobilenetv3_large_100?</p>
</blockquote>
</aside>
<p>Probably the most practical approach would be to rewrite your U-Net<br>
to use MobileNet’s “depthwise separable convolutions” and follow its<br>
structure in how much it downsamples in each block.</p>
<p>The issue is that you have to hold on to references to the outputs of the<br>
downsampling blocks so that you can <code>cat()</code> them together with the<br>
upsampled tensors (U-Net’s “cross connections”) as you work up through<br>
the upsampling blocks.</p>
<p><em>Hypothetically,</em> you could instantiate a <code>MobileNetV3</code> and perform runtime<br>
“surgery”, but I don’t see a clear path to performing all of the necessary<br>
steps, and, even if you could do so, it seems like writing a MobileNet-based<br>
U-Net would be less of a pain in the neck than the surgery approach.</p>
<p>You don’t say anything about using a pretrained model, but if you write<br>
your own model and are careful about keeping the same names for all<br>
of the layers, you ought to be able to load a pretrained <code>state_dict</code> from<br>
a pretrained MobileNet into your model.</p>
<p>(If you were to succeed with the runtime surgery approach, you could<br>
simply perform the surgery on a pretrained MobileNet.)</p>
<aside class="quote no-group">
<blockquote>
<p>Also, the images I wish to train it on are grayscale.</p>
</blockquote>
</aside>
<p>The original MobileNet starts of with a <code>Conv2d</code> with <code>in_channels = 3</code>,<br>
corresponding to an RGB input.  Simply change <code>in_channels</code> to <code>1</code> for<br>
a grayscale input.  (Alternatively, you could simply <code>expand()</code> your inputs<br>
to have three channels, giving you an RGB input that is “grayscale”<br>
because it has R = G = B.)</p>
<p>As an aside, both MobileNet and your U-Net use padding (so that the<br>
outputs of the convolutions have the same spatial extent as their inputs).<br>
The original U-Net very specifically <em>does not</em> do this so as to keep the<br>
“images” on either end of the cross connections properly aligned, avoiding<br>
border artifacts when they are <code>cat()</code>ed together.  You might consider<br>
following the original U-Net strategy in this regard.</p>
<p>Best.</p>
<p>K. Frank</p>