Change dataset from cityscapes to binary segmentation

rui_zhou: <p>Hi, when I try to change the dataset Cityscapes to a binary segmentation, the cross_entropy loss seems not work well with the binary mask,</p>
<blockquote>
<p>Blockquote<br>
File “source_only.py”, line 126, in main<br>
lr_scheduler, epoch, visualize if args.debug else None, args)<br>
File “source_only.py”, line 188, in train<br>
loss_cls_s = criterion(pred_s, label_s)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py”, line 889, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py”, line 1048, in forward<br>
ignore_index=self.ignore_index, reduction=self.reduction)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py”, line 2693, in cross_entropy<br>
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py”, line 2390, in nll_loss<br>
ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)<br>
RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 4</p>
</blockquote>
<p>and if I change it to BCEloss,</p>
<blockquote>
<p>Blockquote<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py”, line 613, in forward<br>
return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py”, line 2755, in binary_cross_entropy<br>
“Please ensure they have the same size.”.format(target.size(), input.size())<br>
ValueError: Using a target size (torch.Size([2, 256, 512, 3])) that is different to the input size (torch.Size([2, 19, 256, 512])) is deprecated. Please ensure they have the same size.</p>
</blockquote>
<p>it goes wrong with the channels, even I  set the num_class from 19 to 2.</p>
<p>I have no idea, any guidance will be so helpful!</p>

ptrblck: <p><code>nn.CrossEntropyLoss</code> used for a multi-class segmentation use case expects a model output in the shape <code>[batch_size, nb_classes, height, width]</code> while the target should have the shape <code>[batch_size, height, width]</code> and contain class indices in the range <code>[0, nb_classes-1]</code> (<code>[0, 1]</code> in your case for a “binary multi-class segmentation”).<br>
The first error is thus raised, since the target seems to have 4 dimensions.</p>
<p>On the other hand, <code>nn.BCEWithLogitsLoss</code> expects both the model output and target to have the same shape as <code>[batch_size, nb_classes, height, width]</code> and the target should contain floating point values in the range <code>[0, 1]</code> for each class (<code>nb_classes</code> would be <code>1</code> for a binary segmentation use case).</p>
<p>Based on the second error message it seems you are trying to pass color images in the channels-last memory format as the target tensors, which won’t work. In case my assumption is correct you would have to map the colors to class values first.</p>

rui_zhou: <p>Hi，I change the map to</p>
<blockquote>
<p>Blockquote<br>
self.id_to_trainid = {<br>
0: 0,<br>
1: 1<br>
}<br>
self.trainid2name = {<br>
0:“back”,<br>
1:“file”,<br>
}</p>
</blockquote>
<p>but it seems the same problem…</p>
<blockquote>
<p>Blockquote<br>
File “train_src.py”, line 314, in <br>
main()<br>
File “train_src.py”, line 307, in main<br>
model = train(cfg, args.local_rank, args.distributed)<br>
File “train_src.py”, line 139, in train<br>
loss = criterion(pred, src_label)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py”, line 889, in _call_impl<br>
result = self.forward(*input, **kwargs)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/modules/loss.py”, line 1048, in forward<br>
ignore_index=self.ignore_index, reduction=self.reduction)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py”, line 2693, in cross_entropy<br>
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)<br>
File “/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py”, line 2390, in nll_loss<br>
ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)<br>
RuntimeError: 1only batches of spatial targets supported (3D tensors) but got targets of size: : [2, 3, 720, 1280]</p>
</blockquote>

ptrblck: <p>I don’t know where this mapping is used, but based on the error message your target still contains 4 dimensions (now the channel dimension seems to be in <code>dim1</code>).<br>
Could you show how you’ve tried to map the color values to the class indices in the target tensor?</p>

rui_zhou: <p>Thanks for reply, the map is in this way:</p>
<blockquote>
<p>class GTA5DataSet(data.Dataset):<br>
def <strong>init</strong>(self,<br>
max_iters=None,<br>
num_classes=2,<br>
ignore_label=255,<br>
debug=False,):</p>
<pre><code>    self.split = split
    self.NUM_CLASS = num_classes
    self.data_root = data_root
    self.data_list = []
    
    if max_iters is not None:
        self.label_to_file, self.file_to_label = pickle.load(open(osp.join(data_root, "gtav_label_info.p"), "rb"))
        self.img_ids = []
        SUB_EPOCH_SIZE = 3000
        tmp_list = []
        ind = dict()
        for i in range(self.NUM_CLASS):
            ind[i] = 0
        for e in range(int(max_iters/SUB_EPOCH_SIZE)+1):
            cur_class_dist = np.zeros(self.NUM_CLASS)
            for i in range(SUB_EPOCH_SIZE):
                if cur_class_dist.sum() == 0:
                    dist1 = cur_class_dist.copy()
                else:
                    dist1 = cur_class_dist/cur_class_dist.sum()
                w = 1/np.log(1+1e-2 + dist1)
                w = w/w.sum()
                c = np.random.choice(self.NUM_CLASS, p=w)

                if ind[c] &gt; (len(self.label_to_file[c])-1):
                    np.random.shuffle(self.label_to_file[c])
                    ind[c] = ind[c]%(len(self.label_to_file[c])-1)

                c_file = self.label_to_file[c][ind[c]]
                tmp_list.append(c_file)
                ind[c] = ind[c]+1
                cur_class_dist[self.file_to_label[c_file]] += 1

        self.img_ids = tmp_list

    if max_iters is not None:
        self.data_list = self.data_list * int(np.ceil(float(max_iters) / len(self.data_list)))
    
    print('length of gta5', len(self.data_list))

    self.id_to_trainid = {0: 0, 1: 1}
    self.trainid2name = {
        0:"back",
        1:"file",
         }
    self.transform = transform
    self.ignore_label = ignore_label
    self.debug = debug

def __len__(self):
    return len(self.data_list)

def __getitem__(self, index):
    if self.debug:
        index = 0
    datafiles = self.data_list[index]

    # re-assign labels to match the format of Cityscapes
    label_copy = self.ignore_label * np.ones(label.shape, dtype=np.float32)
    for k, v in self.id_to_trainid.items():
        label_copy[label == k] = v
    label = Image.fromarray(label_copy)

    if self.transform is not None:
        image, label = self.transform(image, label)

    return image, label, name
</code></pre>
</blockquote>