Debugging Broken UNet Implementation - Can't Overfit

username: JosephC
cooked: <p>I read through <a href="https://discuss.pytorch.org/t/3d-unet-implementation-doesnt-overfit/60669">a related thread</a> and didn’t quite get to an answer so I thought I’d try and raise this issue on my own:</p>
<p>I tried implementing a UNet for image-to-image learning.  To debug it I’m trying to overfit on a single example.  It learns most of the image perfectly well, but the area where the convolutions seem to overlap in the final image are encountering problems.  I’m not sure where my issue is.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/9/29338bca9b8f4f7d04db251766b8902229e4474d.png" data-download-href="https://discuss.pytorch.org/uploads/default/29338bca9b8f4f7d04db251766b8902229e4474d" title="chrome_2021-12-12_15-28-27"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29338bca9b8f4f7d04db251766b8902229e4474d_2_326x500.png" alt="chrome_2021-12-12_15-28-27" data-base62-sha1="5Su0ON5HrjsWWIXENwWuwCEXctL" width="326" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29338bca9b8f4f7d04db251766b8902229e4474d_2_326x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/9/29338bca9b8f4f7d04db251766b8902229e4474d_2_489x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/9/29338bca9b8f4f7d04db251766b8902229e4474d.png 2x" data-dominant-color="B6B5A4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">chrome_2021-12-12_15-28-27</span><span class="informations">546×836 266 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>One can see the top (output now) versus the bottom (as it should be).</p>
<p>I have a full implementation and the related training code uploaded here: <a href="https://github.com/JosephCatrambone/UNet" class="inline-onebox" rel="noopener nofollow ugc">GitHub - JosephCatrambone/UNet</a></p>
<p>But here is the relevant model architecture:</p>
<pre><code class="lang-auto">class DoubleConv(nn.Module):
	def __init__(self, in_channels: int, out_channels: int):
		super(DoubleConv, self).__init__()
		self.op = nn.Sequential(
			# No bias 'cause we're using BatchNorm.  It will get cancelled out.
			nn.Conv2d(in_channels, out_channels, kernel_size=3, stride=1, padding=1, bias=False),
			nn.BatchNorm2d(out_channels),
			nn.LeakyReLU(inplace=True),

			nn.Conv2d(out_channels, out_channels, kernel_size=3, stride=1, padding=1, bias=False),
			nn.BatchNorm2d(out_channels),
			#nn.LeakyReLU(inplace=True),
		)

	def forward(self, x):
		return self.op(x)

class UNet(nn.Module):
	def __init__(self, in_channels:int = 3, out_channels: int = 3, feature_counts=None):
		super(UNet, self).__init__()

		# Prevent modification of mutable default.
		if feature_counts is None:
			feature_counts = [64, 128, 256, 512]

		# Need nn.ModuleList instead of List for batch evals.
		self.downsamples = nn.ModuleList()
		self.bottleneck = DoubleConv(feature_counts[-1], feature_counts[-1]*2)
		self.upsamples = nn.ModuleList()
		self.finalconv = nn.Conv2d(feature_counts[0], out_channels, kernel_size=1)  # 1x1 conv -&gt; Change # feats.
		self.pool = nn.MaxPool2d(kernel_size=2, stride=2)

		# Downsample-Reduction step.
		num_channels = in_channels
		for f_count in feature_counts:
			self.downsamples.append(DoubleConv(in_channels=num_channels, out_channels=f_count))
			num_channels = f_count

		# Up-steps.
		for f_count in reversed(feature_counts):
			self.upsamples.append(nn.ConvTranspose2d(f_count*2, f_count, kernel_size=2, stride=2))
			self.upsamples.append(DoubleConv(f_count*2, f_count))

	def forward(self, x):
		skip_connections = list()  # Don't need ModuleList because this is not retained.
		for dwn in self.downsamples:
			x = dwn(x)
			skip_connections.append(x)
			x = self.pool(x)

		x = self.bottleneck(x)
		skip_connections.reverse()

		for idx in range(0, len(self.upsamples), 2):
			x = self.upsamples[idx](x)
			sk = skip_connections[idx//2]

			# It's possible that due to integer division the sizes slightly mismatch.
			if x.shape != sk.shape:
				x = torch.nn.functional.interpolate(x, size=sk.shape[2:])

			assert len(sk.shape) == 4  # So we don't accidentally unpinch another dimension.
			concat_skip = torch.cat((sk, x), dim=1)  # Dim 1 is channel-dimension.  [b, c, h, w]
			x = self.upsamples[idx+1](concat_skip)

		return self.finalconv(x)
</code></pre>
<p>My best guesses for the source of the issue are something to do with the upscaling and the concatenation – maybe I’m joining along the wrong axis.  Or, more likely, there’s something wrong with the padded convolution, but I can’t seem to make it work without the same-size conv2d.</p>
<p>Any input or debugging tips would be appreciated.</p>

username: ptrblck
cooked: <p>I guess the issue is caused by the upsampling layers, which are using transposed convs as well as plain <code>nn.Conv2d</code> layers with overlapping kernels. I would probably try to either remove the <code>nn.Conv2d</code> layers or make sure the kernel do not overlap.</p>

username: JosephC
cooked: <p>Thank you for the input!</p>
<p>I tried swapping out the Conv2dTranspose with an Upsample2D/Linear Interpolation and I’m seeing the same artifact.  :’(</p>
<p>The network is pretty much all Conv2ds, so I don’t think I can really ‘remove them’?  And WRT overlap,  I thought that kernels <em>have</em> to be able to overlap for convolution to be useful?  There are some articles on it that seems to imply it’s okay: <a href="https://distill.pub/2016/deconv-checkerboard/" class="inline-onebox" rel="noopener nofollow ugc">Deconvolution and Checkerboard Artifacts</a></p>
<p>Other things I thought might have been problems: BatchNorm with bias=False.  Normally for overfitting the bias channel is the thing that takes up the brunt of the effort.  By setting bias to false it forces all of the image data to get encoded in the weights, which for a zero input means that there’s no chance of a decent fit.  I turned off batch norm and it didn’t seem to help.</p>
<p>My next step to debug is going to be replacing the all-zero input with all-one input.  I’m not going to put back the batch norm yet.</p>
<p>If that fails, I’ll try pulling everything but the U part of the network.  I think the depth-convolution might be working on pixel channels instead of depth.  :c  Doesn’t seem like it, but…</p>

username: JosephC
cooked: <p>FINALLY!  After 63 revisions, I figured it out.  The input size for the image I had was too big.  There weren’t enough weights to overfit.  <img src="https://discuss.pytorch.org/images/emoji/apple/man_facepalming.png?v=12" title=":man_facepalming:" class="emoji" alt=":man_facepalming:" loading="lazy" width="20" height="20"> I went back to formula and dropped the width/height by 50% and it was able to overfit in less than 2k iterations.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf17a2832a22c01cc1f636d345e3fa475599f3f2.png" data-download-href="https://discuss.pytorch.org/uploads/default/cf17a2832a22c01cc1f636d345e3fa475599f3f2" title="chrome_2021-12-14_17-25-18"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf17a2832a22c01cc1f636d345e3fa475599f3f2_2_280x500.png" alt="chrome_2021-12-14_17-25-18" data-base62-sha1="ty1tn35bVyZX3jYx3d5xY26nn2O" width="280" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf17a2832a22c01cc1f636d345e3fa475599f3f2_2_280x500.png, https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf17a2832a22c01cc1f636d345e3fa475599f3f2.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf17a2832a22c01cc1f636d345e3fa475599f3f2.png 2x" data-dominant-color="C0C0B3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">chrome_2021-12-14_17-25-18</span><span class="informations">392×698 164 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thank you to everyone that helped (both here and in other forums).  This is a wonderful outcome.</p>

username: ptrblck
cooked: <p>That’s really interesting, as I was looking into a “functional” issue in your code.<br>
Based on your update and the initial output image my current guess was that maybe a reshaping operation might be failing somehow (although I didn’t have an idea where and why it should be used).</p>