White noises in prediction result from an convolution encoder-decoder model

username: zhenkai
cooked: <p>I have been working on a project that outputs monocular depth estimation from a single RGB image. By modifying open-source code on GitHub I manage to obtain results like the following.</p>
<p>[Image 1, see reply below, a new user can only post 1 embedded image in topic =.=‘’]</p>
<p>Note that the result above is a prediction made at resolution = 2560 x 1920 but the model is actually trained on images that are 640 x 480, hence the structural inconsistency shown in the image above. Currently, I want to reduce/eliminate such structural inconsistency from the image and I decided to add another loss into the existing loss function to do so. The existing loss function is:</p>
<pre><code class="lang-auto">class silog_loss(nn.Module):
    def __init__(self, variance_focus):
        super(silog_loss, self).__init__()
        self.variance_focus = variance_focus

    def forward(self, depth_est, depth_gt, mask):
        d = torch.log(torch.abs(depth_est[mask])) - torch.log(depth_gt[mask])
        return (torch.sqrt((d.clone() ** 2).mean() - self.variance_focus * (d.clone().mean() ** 2)) * 10.0)
</code></pre>
<p>which is a standard regression loss in log space. Note that depth_est, depth_gt are tensors of dimension (Batchsize, Channel, Width, Height) = (8, 1, 640, 480).</p>
<p>I added another multiscale loss into the regression loss above using the code below:</p>
<pre><code class="lang-auto">class MSG_loss(nn.Module):
    def __init__(self):
        super(MSG_loss, self).__init__()

    def forward(self, depth_est, depth_gt, mask):
        depth_est = depth_est * mask.to(torch.float)
        depth_gt = depth_gt * mask.to(torch.float)
        gradient_loss = imgrad_loss(depth_est, depth_gt)

        depth_est = torch_nn_func.interpolate(depth_est, scale_factor=(0.5, 0.5), mode='nearest')
        depth_gt = torch_nn_func.interpolate(depth_gt, scale_factor=(0.5, 0.5), mode='nearest')
        gradient_loss2 = imgrad_loss(depth_est, depth_gt)

        depth_est = torch_nn_func.interpolate(depth_est, scale_factor=(0.5, 0.5), mode='nearest')
        depth_gt = torch_nn_func.interpolate(depth_gt, scale_factor=(0.5, 0.5), mode='nearest')
        gradient_loss3 = imgrad_loss(depth_est, depth_gt)

        depth_est = torch_nn_func.interpolate(depth_est, scale_factor=(0.5, 0.5), mode='nearest')
        depth_gt = torch_nn_func.interpolate(depth_gt, scale_factor=(0.5, 0.5), mode='nearest')
        gradient_loss4 = imgrad_loss(depth_est, depth_gt)

        depth_est = torch_nn_func.interpolate(depth_est, scale_factor=(0.5, 0.5), mode='nearest')
        depth_gt = torch_nn_func.interpolate(depth_gt, scale_factor=(0.5, 0.5), mode='nearest')
        gradient_loss5 = imgrad_loss(depth_est, depth_gt)

        return (0.005 * (gradient_loss + gradient_loss2 + gradient_loss3 + gradient_loss4 + gradient_loss5))
</code></pre>
<pre><code class="lang-auto">def imgrad_loss(pred, gt):

    grad_y, grad_x = imgrad(pred)
    grad_y_gt, grad_x_gt = imgrad(gt)
    grad_y_diff = torch.abs(grad_y - grad_y_gt)
    grad_x_diff = torch.abs(grad_x - grad_x_gt)
    # if mask is not None:
    #     grad_y_diff = grad_y_diff[mask]
    #     grad_x_diff = grad_x_diff[mask]
    return (torch.mean(grad_y_diff) + torch.mean(grad_x_diff))

def imgrad(img):
    img = torch.mean(img, 1, True)
    fx = np.array([[1,0,-1],[2,0,-2],[1,0,-1]])
    
    convolution1 = nn.Conv2d(1, 1, kernel_size=3, stride=1, padding=1, bias=False)
    weight = torch.from_numpy(fx).float().unsqueeze(0).unsqueeze(0)
    if img.is_cuda:
        weight = weight.cuda()
    convolution1.weight = nn.Parameter(weight)
    grad_x = convolution1(img)

    fy = np.array([[1,2,1],[0,0,0],[-1,-2,-1]])
    convolution2 = nn.Conv2d(1, 1, kernel_size=3, stride=1, padding=1, bias=False)
    weight = torch.from_numpy(fy).float().unsqueeze(0).unsqueeze(0)

    if img.is_cuda:
        weight = weight.cuda()

    convolution2.weight = nn.Parameter(weight)
    grad_y = convolution2(img)

    return grad_y, grad_x
</code></pre>
<p>As shown from the code, I multiplied the multi-scale gradient loss with <em><strong>0.005</strong></em> and the prediction result returns a depth map with several white noises as follow (not very obvious, will have to zoom in to see the white dots)</p>
<p>[Image 2, see reply below, a new user can only post 1 embedded image in topic =.=‘’]</p>
<p>When the multi-scale gradient loss is multiplied with <em><strong>0.1</strong></em>, it yields results like the following:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/7/f729bc3b1dad991c9560de85a4aecd62aca79f80.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f729bc3b1dad991c9560de85a4aecd62aca79f80" title="sample20"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f729bc3b1dad991c9560de85a4aecd62aca79f80_2_666x500.jpeg" alt="sample20" data-base62-sha1="zgvlRt7oDoov8NQdTRnOLBG9mjm" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f729bc3b1dad991c9560de85a4aecd62aca79f80_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f729bc3b1dad991c9560de85a4aecd62aca79f80_2_999x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/7/f729bc3b1dad991c9560de85a4aecd62aca79f80_2_1332x1000.jpeg 2x" data-dominant-color="D2D2D2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">sample20</span><span class="informations">1920×1440 369 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>To summarize, when the proportion of the multi-scale gradient is high, the output of the model gives a nice prediction with clear edges and better structural consistency but with several white lines in the image (The loss is working? it does have a positive impact on the model).</p>
<p>Reducing the proportion of the multi-scale gradient reduces the white lines but white dots appear and the improvement in structural consistency is minimal  (Reducing the negative impact of the loss but the positive impact is gone in the meantime as well).</p>
<p>I am continuing the training while further lowering the proportion of the multi-scale gradient loss but I doubt it will have any improvement in my model when the white dots are fully eliminated. I am stuck here right now and any suggestions/advice is greatly appreciated!!</p>

username: zhenkai
cooked: <p>Image 1</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/8/6817b47d75ec633a170e849eca2e8312b2d94b54.png" alt="image" data-base62-sha1="eQQrzzPS3JkvnyfKPNbm1ZSiB1i" width="170" height="130"></p>

username: zhenkai
cooked: <p>Image 2<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/3/d3af810ae2f6c2396b50b178c128d06de14eeda3.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/d3af810ae2f6c2396b50b178c128d06de14eeda3" title="sample20"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d3af810ae2f6c2396b50b178c128d06de14eeda3_2_666x500.jpeg" alt="sample20" data-base62-sha1="ucELf9hcgdUuBpWlgrzWhIhZIgH" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d3af810ae2f6c2396b50b178c128d06de14eeda3_2_666x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d3af810ae2f6c2396b50b178c128d06de14eeda3_2_999x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/3/d3af810ae2f6c2396b50b178c128d06de14eeda3_2_1332x1000.jpeg 2x" data-dominant-color="535353"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">sample20</span><span class="informations">1920×1440 208 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>