Encoder-Bottleneck-Decoder Unet Based Semantic Segmentation

Idrees11: <p>I have Gray Image Size as: 256 x 256 . I  am using Four convolutional Layers with 96 filters per layer. However, I am getting error: Please see my code</p>
<pre><code class="lang-auto">#-----------------------------------------------------------------------------#
#-----------------------------------------------------------------------------#
import torch
import torch.nn as nn
import cv2
import torch
from torchvision import transforms
from torchviz import make_dot
import numpy as np
#-----------------------------------------------------------------------------#
#-----------------------------------------------------------------------------#
class conv_block(nn.Module):
    
    def __init__(self, in_c, out_c, kernel_size=3):
        super().__init__()
        self.conv1 = nn.Conv2d(in_c, out_c, kernel_size, padding=0)
        self.in1   = nn.InstanceNorm2d(out_c)
        self.relu  = nn.ReLU(inplace=True)
        
    def forward(self, inputs):
        x = self.conv1(inputs)
        x = self.in1(x)
        x = self.relu(x)
        return x
    
class encoder_block(nn.Module):
    
    def __init__(self, in_c, out_c):
        super().__init__()
        self.conv = conv_block(in_c, out_c)
        self.pool = nn.MaxPool2d((2, 2))
        
    def forward(self, inputs):
        x = self.conv(inputs)
        p = self.pool(x)
        return x, p

class decoder_block(nn.Module):
    
    def __init__(self, in_c, out_c):
        super().__init__()
        self.up = nn.ConvTranspose2d(in_c, out_c, kernel_size=3, stride=2, padding=0)
        self.conv = conv_block(out_c+ out_c, out_c)
        
    def forward(self, inputs, skip):
        x = self.up(inputs)
        x = torch.cat([x, skip], axis=1)
        x = self.conv(x)
        return x
        
    
class build_unet(nn.Module):
    
    def __init__(self):
        super().__init__()
#-----------------------------------------------------------------------------#
#                        ENCODER                                              #
#-----------------------------------------------------------------------------#
        self.e1 = encoder_block(1 ,96)
        self.e2 = encoder_block(96,96)
        self.e3 = encoder_block(96,96)
        self.e4 = encoder_block(96,96)
#-----------------------------------------------------------------------------#
#                 BOTTLENECK                                                  #
#-----------------------------------------------------------------------------#
        self.b = conv_block(96,96,1)
#-----------------------------------------------------------------------------#
#                   DECODER                                                   #
#-----------------------------------------------------------------------------#
        self.d1 = decoder_block(96,96)
        self.d2 = decoder_block(96,96)
        self.d3 = decoder_block(96,96)
        self.d4 = decoder_block(96,3)
#----------------------------------------------------#
#              """ Classifier """
#----------------------------------------------------#
        # self.outputs = nn.Conv2d(1, 96, kernel_size=1, padding=1)
#------------------------------------------------------------------------------#
#                   """ Encoder """
#-----------------------------------------------------------------------------#        
    def forward(self, inputs):
        s1, p1 = self.e1(inputs)
        s2, p2 = self.e2(p1)
        s3, p3 = self.e3(p2)
        s4, p4 = self.e4(p3)
#------------------------------------------------------------------------------#
#                     """ Bottleneck """
#-----------------------------------------------------------------------------#
        b = self.b(p4)
#------------------------------------------------------------------------------#
#                     """ Decoder """
#------------------------------------------------------------------------------#
        d1 = self.d1(b, s4)
        d2 = self.d2(d1, s3)
        d3 = self.d3(d2, s2)
        d4 = self.d4(d3, s1)
        return d4
#-------------------------------------------#
#          """ Classifier """               #
#-------------------------------------------#
        # outputs = self.outputs(d4)
        # return outputs
#-------------------------------------------------------------------------------#

if __name__ == "__main__":
    # inputs = torch.randn([1,1,256,256])
    image = cv2.imread (r'C:\Users\Idrees Bhat\Desktop\Research\Insha\Dataset\o_gray\1.png',0)
    convert_tensor = transforms.ToTensor()
    inputs=convert_tensor(image)
    # print (inputs.shape)
    # print (type(inputs)) # [1,256,256]
    model = build_unet()
    y = model (inputs)
    # print(y.shape)type or paste code here
</code></pre>
<p><span class="hashtag-raw">#-----------------------------------------------------------------------------------------------------</span>------------------<br>
ERROR:<br>
File ~\AppData\Local\anaconda3\Lib\site-packages\spyder_kernels\py3compat.py:356 in compat_exec<br>
exec(code, globals, locals)</p>
<p>File c:\users\idrees bhat\desktop\insha\generator_aiwa.py:113<br>
y = model (inputs)</p>
<p>File ~\AppData\Local\anaconda3\Lib\site-packages\torch\nn\modules\module.py:1518 in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)</p>
<p>File ~\AppData\Local\anaconda3\Lib\site-packages\torch\nn\modules\module.py:1527 in _call_impl<br>
return forward_call(*args, **kwargs)</p>
<p>File c:\users\idrees bhat\desktop\insha\generator_aiwa.py:93 in forward<br>
d1 = self.d1(b, s4)</p>
<p>File ~\AppData\Local\anaconda3\Lib\site-packages\torch\nn\modules\module.py:1518 in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)</p>
<p>File ~\AppData\Local\anaconda3\Lib\site-packages\torch\nn\modules\module.py:1527 in _call_impl<br>
return forward_call(*args, **kwargs)</p>
<p>File c:\users\idrees bhat\desktop\insha\generator_aiwa.py:47 in forward<br>
x = torch.cat([x, skip], axis=1)</p>
<p>RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 29 but got size 28 for tensor number 1 in the list.</p>

KFrank: <p>Hi Idrees!</p>
<aside class="quote no-group quote-modified" data-username="Idrees11" data-post="1" data-topic="214096" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/idrees11/48/74448_2.png" class="avatar"> Idrees11:</div>
<blockquote>
<p>I have Gray Image Size as: 256 x 256 .<br>
…<br>
File c:\users\idrees bhat\desktop\insha\generator_aiwa.py:47 in forward<br>
x = torch.cat([x, skip], axis=1)</p>
<p>RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 29 but got size 28 for tensor number 1 in the list.</p>
</blockquote>
</aside>
<p>I haven’t looked at your code in detail, but I believe what I am saying is correct.</p>
<p>There are two issues:</p>
<p>First, as your error message is telling you, the tensor from the skip connection<br>
from the “downward.” contracting side of the U doesn’t have the same spatial<br>
size as the tensor you are trying to <code>cat()</code> it with on the “upward,” expansive<br>
side of the U.  Following the original U-Net architecture, you need to crop the<br>
skip-connection tensor to match the size of the “upward” tensor.</p>
<p>But there is some nuance to this.</p>
<p>First, you are using <code>padding = 0</code> in your <code>Conv2d</code>s and <code>ConvTranspose2d</code>s.<br>
This is correct (so leave it that way).</p>
<p>As your input tensor passes through the U-Net, its spatial size is being changed<br>
in three ways:  The convolutions (with a <code>3x3</code> kernel and no padding) trim a pixel<br>
off of each edge.  That is, each spatial dimension is reduced by 2.  So your image<br>
shrinks going both down and up the U.  Your <code>MaxPool2d</code> layers reduce the<br>
spatial size (of both dimensions) by a factor of two going down the U and the<br>
<code>stride = 2</code> <code>ConvTranspose2d</code> layers increase the size of the image by a<br>
compensating factor of two going back up the U.</p>
<p>But because of the trimming, your skip-connection tensors are larger than than<br>
the corresponding “upward” tensors, leading to the need to crop.</p>
<p>More subtly, the input to <em>each</em> <code>MaxPool2d</code> needs to have spatial sizes that are<br>
multiples of two (so that the size can be reduced by a factor of two without any<br>
misalignment or interpolation).  This condition restricts the U-Net to accept input<br>
tensors of specific spatial sizes.</p>
<p>To illustrate this, when you pass a <code>256x256</code> image to your U-Net, the first<br>
<code>Conv2d</code> trims the image to <code>254x254</code> and the first <code>MaxPool2d</code> layer divides it<br>
in two to <code>127x127</code>.  Then the second <code>Conv2d</code> trims it two <code>125x125</code>, but these<br>
spatial extents are not factors of two, so subsequent <code>MaxPool2d</code>s can’t be<br>
performed cleanly without introducing artifacts.  (Attempts to fix this up by<br>
cropping or interpolating would also introduce artifacts.)</p>
<p>The sound way to address this is to work through your specific U-Net architecture<br>
layer by layer to determine  the “legal” input sizes <em>and</em> their corresponding output<br>
sizes.  Pad your ground-truth <code>target</code> data to the next largest legal output size, find<br>
the legal input size that corresponds to that output size and reflection-pad your<br>
input tensor to that legal size.</p>
<p>Lastly, you would typically use a per-pixel loss criterion for training (for example,<br>
<code>BCEWithLogitsLoss</code>).  It’s best to include in the loss only contributions from the<br>
pixels in the original <code>target</code> tensor, ignoring the padded pixels.</p>
<p>“But,” you say, “this is all very complicated!”  Yes, but it’s the right thing to do.<br>
The inventors of U-Net made sure to get ti right in their original U-Net paper,<br>
so you can too.</p>
<p>Best.</p>
<p>K. Frank</p>

Idrees11: <p>Thank you Sir for your encouragement</p>