Checkerborad effect

blackbirdbarber: <p>I slightly modified the <a href="https://github.com/ptrblck/pytorch_misc/blob/master/unet_demo.py" rel="noopener nofollow ugc">Unet</a> and now I experience this<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/3/e3de450acc7985d97f9e29967d295a8b0f26fb18.jpeg" alt="image" data-base62-sha1="wvOAYDrB0ONZobfWwhb85m9I9SM" width="296" height="204"><br>
for the GT of</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a927edd845b2115876eb1b83f4c1e3caf202440c.png" alt="image" data-base62-sha1="o8qeTcLDUKSCDj4NyNoWX9Viq8k" width="306" height="214"></p>
<p>I assume this is due to ConvTranspose2D. How to overcome this?</p>
<pre><code class="lang-auto">class UpConv(nn.Module):
    def __init__(self, in_channels, in_channels_skip, out_channels,
                 kernel_size, padding, stride):
        super(UpConv, self).__init__()
        
        self.act = nn.ReLU()

        self.conv_trans1 = nn.ConvTranspose2d(in_channels, in_channels, kernel_size=2, padding=0, stride=2)
        
        self.bn1 = nn.BatchNorm2d(in_channels, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        self.bn2 = nn.BatchNorm2d(out_channels, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)

        self.conv_block = BaseConv(
            in_channels=in_channels + in_channels_skip,
            out_channels=out_channels,
            kernel_size=kernel_size,
            padding=padding,
            stride=stride)

    def forward(self, x, x_skip):
        x = self.act(self.bn1(self.conv_trans1(x)))
       
        x = torch.cat ((x, x_skip[:, :, :x.shape[2], :x.shape[3]]), dim = 1)
        
        x = self.act(self.bn2(self.conv_block(x)))
        return x
</code></pre>
<p>I am using always 400x400 images to train so the slicing is really not that needed <code>x</code> and <code>x_skip</code> do match for the <code>cat</code> always.</p>
<p><a href="https://pastebin.com/RBDd1xdB" rel="noopener nofollow ugc">source code</a></p>

KFrank: <p>Hi Blackbird!</p>
<aside class="quote no-group" data-username="blackbirdbarber" data-post="1" data-topic="158761" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/6de8d8/48.png" class="avatar"> blackbirdbarber:</div>
<blockquote>
<p>I am using always 400x400 images to train so the slicing is really not that needed <code>x</code> and <code>x_skip</code> do match for the <code>cat</code> always.</p>
</blockquote>
</aside>
<p>Speaking specifically of the architecture of the original U-Net paper (so<br>
this comment may or may not be relevant to your use case), in which<br>
the convolutions are not padded and they nibble away at the edges of<br>
the image, a size of <code>400</code> will not produce an <code>x</code> and <code>x_skip</code> whose image<br>
dimensions match (as would be required for <code>cat()</code> to succeed).</p>
<p>The nearest “self-consistent” sizes to <code>400</code> (for the input image) are <code>396</code><br>
and <code>412</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

blackbirdbarber: <p>Hi Frank,</p>
<p>Valuable feedback of yours and appreciated more than highly.</p>
<p>This architecture only works with image sizes that are 8*x.</p>
<pre><code class="lang-auto">
shapes:x,x_skip torch.Size([1, 512, 100, 100]) torch.Size([1, 256, 100, 100])

shapes:x,x_skip torch.Size([1, 256, 200, 200]) torch.Size([1, 128, 200, 200])

shapes:x,x_skip torch.Size([1, 128, 400, 400]) torch.Size([1, 64, 400, 400])

</code></pre>
<p>Are you suggesting channel dimension of <code>x</code> should match the channel dimension of <code>x_skip</code>, or the <em>upsampled output</em> channels should match <em>high resolution features</em> channels (<em>terms from the paper</em>).</p>
<p>I am not using mirroring, you are right just the simple zero padding of 1 in size to compensate. I know the zero padding is evil, and I would probably benefit from mirror padding but let’s pretend this is not causing the checkerboard effects.</p>
<p>Unless you are referring to the references of the original U-Net paper I haven’t found any evidence after reading it carefully that channel dimensions for <code>x</code> and <code>x_skip</code> should match for the <code>cat</code>.</p>
<p>Do you advise that this would be beneficial? That would be needed of course if you would be using the Linknet architecture.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2cda003194b02d70f8a452dc11fb2f5e4978da73.png" data-download-href="https://discuss.pytorch.org/uploads/default/2cda003194b02d70f8a452dc11fb2f5e4978da73" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2cda003194b02d70f8a452dc11fb2f5e4978da73_2_345x107.png" alt="image" data-base62-sha1="6oM4xfDSwsdIpwoewDGbAt5Ghq3" width="345" height="107" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2cda003194b02d70f8a452dc11fb2f5e4978da73_2_345x107.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2cda003194b02d70f8a452dc11fb2f5e4978da73_2_517x160.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/c/2cda003194b02d70f8a452dc11fb2f5e4978da73_2_690x214.png 2x" data-dominant-color="EDEEF0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">796×247 71.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I am tinkering upon your idea, give me some clue or evidence why this may be important.</p>

blackbirdbarber: <p><a class="mention" href="/u/kfrank">@KFrank</a> after <a href="https://distill.pub/2016/deconv-checkerboard/" rel="noopener nofollow ugc">some checking</a> it looks that stride 2 convolution or stride 2 transposed convolutions will produce the checkerboard effect no matter if the kernel size is divisible by the stride.</p>
<p>This led me to the conclusion to avoid them and to use the resize option since even max pooling provides high frequency image distortions as well.</p>
<p>I wonder if <a class="mention" href="/u/ptrblck">@ptrblck</a> has something to add from his victorious experience since I modified his U-Net  <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=12" title=":thinking:" class="emoji" alt=":thinking:" loading="lazy" width="20" height="20"></p>

KFrank: <p>Hi Blackbird!</p>
<aside class="quote no-group" data-username="blackbirdbarber" data-post="4" data-topic="158761" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/6de8d8/48.png" class="avatar"> blackbirdbarber:</div>
<blockquote>
<p>it looks that stride 2 convolution or stride 2 transposed convolutions will produce the checkerboard effect no matter if the kernel size is divisible by the stride.</p>
</blockquote>
</aside>
<p>I don’t have any concrete evidence that contradicts your conclusion, but<br>
I don’t find the argument in the online article you cite compelling on this<br>
point.</p>
<p>Quoting from that article:</p>
<blockquote>
<p>One approach is to make sure you use a kernel size that is divided by<br>
your stride, avoiding the overlap issue.</p>
</blockquote>
<p>In the context of pytorch’s <code>ConvTranspose2d</code>, this means using, for<br>
example:</p>
<pre data-code-wrap="python"><code class="lang-python">ConvTranspose2d (in_channels, out_channels, kernel_size = 2, stride = 2)
</code></pre>
<p>This is exactly the pytorch-class implementation of the “up-conv 2x2”<br>
described in the original U-Net paper.</p>
<p>As your online paper explains, this specific upsampling does not cause<br>
checkerboarding by the mechanism discussed in some detail in that paper.</p>
<p>However, your paper then goes on:</p>
<blockquote>
<p>However, while this approach helps, it is still easy for deconvolution to<br>
fall into creating artifacts.</p>
</blockquote>
<p>I just don’t know what to make of this statement.  It seems to be saying<br>
that U-Net’s (original) “up-conv 2x2” does (or might) nonetheless cause<br>
artifacts, but it doesn’t say what kinds of artifacts (checkerboarding or<br>
something else) those might be, nor by what mechanism those artifacts<br>
might be caused.</p>
<p>In general I think the online paper you cite is quite good – it gives a good,<br>
understandable explanation, e.g., with its <code>kernel_size = 3</code>, <code>stride = 2</code><br>
example, of how upsampling can produce checkerboarding.  But that<br>
second statement I quoted notwithstanding, it doesn’t show how U-Net’s<br>
“up-conv 2x2” would produce checkerboarding, and, in fact, outlines an<br>
argument to the contrary.</p>
<p>Best.</p>
<p>K. Frank</p>

blackbirdbarber: <p>There is no checkerboard effect in the new architecture with stride 1.</p>
<p>For the GT<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b3c8b59d371bc4eec2060a6b3c9c6a85312ec24e.png" alt="image" data-base62-sha1="pEru52hI3HhLUAN9I63Y2E8EcGO" width="239" height="205"></p>
<p>I am getting<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c75d995a5e07613cdc4bb18d27bafbaf12c0d87.png" alt="image" data-base62-sha1="6ljv3XJwIm67WaDZ1Sy4It2H7Cf" width="236" height="210"></p>