UNET - Upsample vs. ConvTranspose2d

xavier88: <p>Hello.</p>
<p>I am trying to use UNET for my project to find different animals from the pictures. It’s working fine, but I would like to get better accuracy.<br>
I Googled around and found some posts that using ConvTranspose2d causes so called checkerboard pattern of artifacts. Other people have suggested to use Upsample. Tried to implement that.</p>
<p>The initial code is this:<br>
class DoubleConv(nn.Module):<br>
def <strong>init</strong>(self, in_channels, out_channels):<br>
super(DoubleConv, self).<strong>init</strong>()<br>
self.conv = nn.Sequential(<br>
nn.Conv2d(in_channels, out_channels, 3, 1, 1, bias=False),<br>
nn.BatchNorm2d(out_channels),<br>
nn.ReLU(inplace=True),<br>
nn.Conv2d(out_channels, out_channels, 3, 1, 1, bias=False),<br>
nn.BatchNorm2d(out_channels),<br>
nn.ReLU(inplace=True),<br>
)</p>
<pre><code>def forward(self, x):
    return self.conv(x)
</code></pre>
<p>class UNET(nn.Module):<br>
def <strong>init</strong>(<br>
self, in_channels=3, num_classes=5, out_channels=1, features=[64, 128, 256, 512],<br>
):<br>
super(UNET, self).<strong>init</strong>()<br>
self.ups = nn.ModuleList()<br>
self.downs = nn.ModuleList()<br>
self.pool = nn.MaxPool2d(kernel_size=2, stride=2)</p>
<pre><code>    for feature in features:
        self.downs.append(DoubleConv(in_channels, feature))
        in_channels = feature

    for feature in reversed(features):
        self.ups.append(
            nn.ConvTranspose2d(
                feature*2, feature, kernel_size=2, stride=2,
            )
        )
        self.ups.append(DoubleConv(feature*2, feature))

    self.bottleneck = DoubleConv(features[-1], features[-1]*2)
    self.final_conv = nn.Conv2d(features[0], out_channels, kernel_size=1)

def forward(self, x):
    skip_connections = []

    for down in self.downs:
        x = down(x)
        skip_connections.append(x)
        x = self.pool(x)

    x = self.bottleneck(x)
    skip_connections = skip_connections[::-1]

    for idx in range(0, len(self.ups), 2):
        x = self.ups[idx](x)
        skip_connection = skip_connections[idx//2]

        if x.shape != skip_connection.shape:
            x = TF.resize(x, size=skip_connection.shape[2:])

        concat_skip = torch.cat((skip_connection, x), dim=1)
        x = self.ups[idx+1](concat_skip)

    return self.final_conv(x)
</code></pre>
<p>Not sure if I changed the code correctly, but after adding Upsample, I am getting CUDA Out of Memory error.<br>
class DoubleConv(nn.Module):<br>
def <strong>init</strong>(self, in_channels, out_channels):<br>
super(DoubleConv, self).<strong>init</strong>()<br>
self.conv = nn.Sequential(<br>
nn.Upsample(scale_factor = 2, mode=‘nearest’),<br>
nn.Conv2d(in_channels, out_channels, 3, 1, 1, bias=False),<br>
nn.BatchNorm2d(out_channels),<br>
nn.ReLU(inplace=True),<br>
nn.Conv2d(out_channels, out_channels, 3, 1, 1, bias=False),<br>
nn.BatchNorm2d(out_channels),<br>
nn.ReLU(inplace=True),<br>
)</p>
<pre><code>def forward(self, x):
    return self.conv(x)
</code></pre>
<p>class UNET(nn.Module):<br>
def <strong>init</strong>(<br>
self, in_channels=3, num_classes=5, out_channels=1, features=[64, 128, 256, 512],<br>
):<br>
super(UNET, self).<strong>init</strong>()<br>
self.ups = nn.ModuleList()<br>
self.downs = nn.ModuleList()<br>
self.pool = nn.MaxPool2d(kernel_size=2, stride=2)</p>
<pre><code>    for feature in features:
        self.downs.append(DoubleConv(in_channels, feature))
        in_channels = feature

    for feature in reversed(features):
        self.ups.append(DoubleConv(feature*2, feature))

    self.bottleneck = DoubleConv(features[-1], features[-1]*2)
    self.final_conv = nn.Conv2d(features[0], out_channels, kernel_size=1)

def forward(self, x):
    skip_connections = []

    for down in self.downs:
        x = down(x)
        skip_connections.append(x)
        x = self.pool(x)

    x = self.bottleneck(x)
    skip_connections = skip_connections[::-1]

    for idx in range(0, len(self.ups), 2):
        x = self.ups[idx](x)
        skip_connection = skip_connections[idx//2]

        if x.shape != skip_connection.shape:
            x = TF.resize(x, size=skip_connection.shape[2:])

        concat_skip = torch.cat((skip_connection, x), dim=1)
        x = self.ups[idx+1](concat_skip)

    return self.final_conv(x)
</code></pre>
<p>Is there anything else I need to include for the Upsample?</p>