Need help to create depth and mask prediction in an image

anup1986: <p>Hi,  I want to predict depth and mask in an image. The depth will be predicted for roads and i want to mask a car in the image. I have implemented UNet for this problem.<br>
The dataloader code is<br>
class DepthDataSet(Dataset):<br>
“”" Dataset for Depth and mask prediction “”"</p>
<p>def <strong>init</strong>(self, conf, fg_bg_dir, mask_dir, depth_dir, transform=None, scale=1):<br>
“”"<br>
Args:<br>
conf = configuration file<br>
image_dir: Directory to images of depth<br>
mask_dir: Directory to mask images of depth<br>
transformation: transformations applied on that image<br>
“”"<br>
self.conf = conf<br>
self.fg_bg_dir = fg_bg_dir<br>
self.mask_dir = mask_dir<br>
self.depth_dir = depth_dir<br>
self.scale = scale<br>
self.ids = [file for file in listdir(fg_bg_dir) if not file.startswith(’.’)]</p>
<p>def <strong>len</strong>(self):<br>
return len(self.ids)</p>
<p><span class="mention">@classmethod</span><br>
def preprocess(cls, pil_img, scale):<br>
w, h = pil_img.size<br>
newW, newH = int(scale * w), int(scale * h)<br>
assert newW &gt; 0 and newH &gt; 0, ‘Scale is too small’<br>
pil_img = pil_img.resize((newW, newH))</p>
<pre><code>  img_nd = np.array(pil_img)

  if len(img_nd.shape) == 2:
      img_nd = np.expand_dims(img_nd, axis=2)

  # HWC to CHW
  img_trans = img_nd.transpose((2, 0, 1))
  if img_trans.max() &gt; 1:
      img_trans = img_trans / 255

  return img_trans
</code></pre>
<p>def <strong>getitem</strong>(self, i):<br>
idx = self.ids[i]<br>
# image_file = glob(self.image_dir + ‘/’+ idx )<br>
# mask_file = glob(self.mask_dir + ‘/’+ idx )<br>
# print(idx)<br>
# print(self.image_dir)<br>
# print(mask_file)</p>
<pre><code># assert len(mask_file) &gt; 1, "No mask found"
# assert len(image_file) &gt; 1, "No image found"

mask = Image.open(self.mask_dir + '/'+ idx)
fg_bg = Image.open(self.fg_bg_dir + '/'+ idx)
depth = Image.open(self.depth_dir + '/'+ idx)
#if image.size != mask.size:
#assert image.size == mask.size
#img = self.preprocess(image, self.scale)
#mask = self.preprocess(mask, self.scale)

return {
        'image': torch.from_numpy(np.array(fg_bg)), 
        'mask': torch.from_numpy(np.array(mask)), 
        'depth': torch.from_numpy(np.array(depth))
       }
</code></pre>
<p>I am feeding the image to the model and checking the loss for mask and depth.<br>
images = images.to(device=self.device, dtype=torch.float)<br>
mask_type = torch.float32 if self.model.n_classes == 1 else torch.long<br>
mask = mask.to(device=device, dtype=mask_type)<br>
depth = depth.to(device=device, dtype=mask_type)</p>
<pre><code>        mask_pred = self.model(images)
        loss_mask = self.criterion(mask_pred, mask)
        loss_depth = self.criterion(mask_pred, depth)
        loss = loss_mask + loss_depth

        train_loss += loss_mask.item() + loss_depth.item()
</code></pre>
<p>My result is not proper , Can you please help me to make this solution better.<br>
Or is it the correct approach to add up the loss against the ground truths.</p>
<p>Thanks,</p>

ptrblck: <p>Which criterion are you using and what ranges are expected in <code>mask</code> and <code>depth</code>?<br>
Also, you shouldn’t call <code>item()</code> on the losses, if you plan to call <code>train_loss.backward()</code>.</p>
<p>PS: Could you format the code by wrapping it into three backticks ``` as it’ll be easier for debugging? <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

anup1986: <p>Hi,<br>
Criterion ------&gt; CrossEntropyLoss.<br>
Input image ---------&gt;<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f3998a42dd140f09644fdc6774cb6020c9ea07b2.jpeg" alt="374" data-base62-sha1="yKYYYiJBBiUs7YY2Th5yAoi4AFQ" width="224" height="224"><br>
mask image --------&gt; <img src="https://discuss.pytorch.org/uploads/default/original/3X/0/7/0722b27948fc6d56f3559054a8c1cec640ed30d1.jpeg" alt="63" data-base62-sha1="117GcjmVPYCXCiQc9wkyCnOF9df" width="224" height="224"><br>
depth image --------&gt; <img src="https://discuss.pytorch.org/uploads/default/original/3X/0/4/046cdd3307d9c51919753395e85bd382d06ef05b.jpeg" alt="3949" data-base62-sha1="D99iObkFwbVBKgTa0bsaoHVdQD" width="224" height="224"></p>
<p>The input image i am forwarding through my model and the mask and depth image i am taking as ground truth.<br>
I am doing loss.backward() not on the train loss.<br>
This is my train function</p>
<pre><code class="lang-auto">def train(self, train_acc):
        self.model.train()
        pbar = tqdm(self.train_loader)
        train_loss = 0
        length = len(self.train_loader)
        print("Length of train loader is {}".format(length))
        device = self.device
        self.model.to(self.device)
        for batch in enumerate(pbar):
            images = batch[1]['image']
            mask = batch[1]['mask']
            depth = batch[1]['depth']
            print(len(images), len(mask), len(depth))
            images = images.to(device=self.device, dtype=torch.float)
            mask_type = torch.float32 if self.model.n_classes == 1 else torch.long
            mask = mask.to(device=device, dtype=mask_type)
            depth = depth.to(device=device, dtype=mask_type)

            mask_pred = self.model(images)
           
            loss_mask = self.criterion(mask_pred, mask)
            loss_depth = self.criterion(mask_pred, depth)
            loss = loss_mask +  loss_depth 

            train_loss += loss_mask.item() + loss_depth.item()
            #writer.add_scalar('Loss/train', train_loss, global_step)

            pbar.set_postfix(**{'loss (batch)': train_loss})
            self.optimizer.zero_grad()

            # Backpropagation
            loss.backward()
            self.optimizer.step()
            self.scheduler.step()
        
            accuracy = 100*(train_loss/length)
            pbar.set_description(desc= f'Loss={train_loss} Accuracy={accuracy:0.2f}')
            train_acc.append(accuracy)
</code></pre>
<p>I don’t know whether it is a correct way or not to add losses for the purpose of depth and mask prediction.<br>
My accuracy also goes to negative.<br>
Please help.<br>
Thanks</p>

ptrblck: <p>Based on the code it seems you are using <code>nn.CrossEntropyLoss</code> and <code>LongTensors</code> for the <code>mask</code> and <code>depth</code>. While this approach seems to be fine for the mask target, I’m not sure it is right for the depth estimation, which seems to be a regression task.<br>
I would recommend to try to keep <code>depth</code> as a <code>FloatTensor</code> and use <code>nn.MSELoss</code> for the <code>loss_depth</code> calculation.</p>

anup1986: <p>ok will try with that, thannks</p>