Handle ground truth pixel values equal to 255 in MaskFormer
Author: surajbijjahalli
Pull request: no

I am trying to use Maskformer for segmenting lesions on the ISIC2018 dataset (https://challenge.isic-archive.com/data/#2018). 
The training images are RGB and the training ground truth annotations are binary .png images where the pixel value of 0 is the background, and the pixel value of 255 denotes the lesion (i.e. the class to be segmented). I remember reading somewhere that pixel values of 255 are automatically ignored in MaskFormer. I am confused as to how to set up the id2label file and the parameters for the preprocessor. i.e. should I set reduce_labels to True and should ignore_index be set to 0 ? 

Comments:
Hi,

Yes MaskFormer's image processor supports the [ignore_index](https://huggingface.co/docs/transformers/main/model_doc/maskformer#transformers.MaskFormerImageProcessor.ignore_index) argument, which indicates which value in the ground truth segmentation maps needs to be ignored. MaskFormer has a "binary mask classification" paradigm, which means that it will convert semantic segmentation maps (i.e. maps where every pixel is labeled with a certain class) to a set of binary masks (one for each class present in the ground truth segmentation map).

So in your case, if the ground truth images indicate with 0 = background, and 255 = class to be segmented, then I'd advise to instantiate `MaskFormerImageProcessor` with `ignore_index=0`. There's no need to set `reduce_labels` to `True`.

Btw it would be cool to push the ISIC2018 dataset to the 🤗 hub, see here for more info on how to do that: https://github.com/huggingface/transformers/tree/main/examples/pytorch/semantic-segmentation#note-on-custom-data

Hi Niels. Thanks so much for the advice. I have already pushed the ISIC dataset to the hub at `surajbijjahalli/ISIC2018`. I have been using your excellent tutorial on maskformer for semantic segmentation. I have successfully created dataloaders without any issues. 
However, I am now experiencing the following error when I try and do a forward pass of the batch through the model. I get the following error:
```
IndexError                                Traceback (most recent call last)

[<ipython-input-23-281a4b0e28fc>](https://localhost:8080/#) in <cell line: 10>()
      8 
      9 # Sanity check the output of the untrained model
---> 10 outputs = model(batch["pixel_values"],class_labels=batch["class_labels"],mask_labels=batch["mask_labels"])
     11 
     12 

10 frames

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _wrapped_call_impl(self, *args, **kwargs)
   1516             return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1517         else:
-> 1518             return self._call_impl(*args, **kwargs)
   1519 
   1520     def _call_impl(self, *args, **kwargs):

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _call_impl(self, *args, **kwargs)
   1525                 or _global_backward_pre_hooks or _global_backward_hooks
   1526                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1527             return forward_call(*args, **kwargs)
   1528 
   1529         try:

[/usr/local/lib/python3.10/dist-packages/transformers/models/maskformer/modeling_maskformer.py](https://localhost:8080/#) in forward(self, pixel_values, mask_labels, class_labels, pixel_mask, output_auxiliary_logits, output_hidden_states, output_attentions, return_dict)
   1912 
   1913         if mask_labels is not None and class_labels is not None:
-> 1914             loss_dict: Dict[str, Tensor] = self.get_loss_dict(
   1915                 masks_queries_logits, class_queries_logits, mask_labels, class_labels, auxiliary_logits
   1916             )

[/usr/local/lib/python3.10/dist-packages/transformers/models/maskformer/modeling_maskformer.py](https://localhost:8080/#) in get_loss_dict(self, masks_queries_logits, class_queries_logits, mask_labels, class_labels, auxiliary_logits)
   1734         auxiliary_logits: Dict[str, Tensor],
   1735     ) -> Dict[str, Tensor]:
-> 1736         loss_dict: Dict[str, Tensor] = self.criterion(
   1737             masks_queries_logits, class_queries_logits, mask_labels, class_labels, auxiliary_logits
   1738         )

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _wrapped_call_impl(self, *args, **kwargs)
   1516             return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1517         else:
-> 1518             return self._call_impl(*args, **kwargs)
   1519 
   1520     def _call_impl(self, *args, **kwargs):

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _call_impl(self, *args, **kwargs)
   1525                 or _global_backward_pre_hooks or _global_backward_hooks
   1526                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1527             return forward_call(*args, **kwargs)
   1528 
   1529         try:

[/usr/local/lib/python3.10/dist-packages/transformers/models/maskformer/modeling_maskformer.py](https://localhost:8080/#) in forward(self, masks_queries_logits, class_queries_logits, mask_labels, class_labels, auxiliary_predictions)
   1170 
   1171         # retrieve the matching between the outputs of the last layer and the labels
-> 1172         indices = self.matcher(masks_queries_logits, class_queries_logits, mask_labels, class_labels)
   1173         # compute the average number of target masks for normalization purposes
   1174         num_masks: Number = self.get_num_masks(class_labels, device=class_labels[0].device)

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _wrapped_call_impl(self, *args, **kwargs)
   1516             return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1517         else:
-> 1518             return self._call_impl(*args, **kwargs)
   1519 
   1520     def _call_impl(self, *args, **kwargs):

[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://localhost:8080/#) in _call_impl(self, *args, **kwargs)
   1525                 or _global_backward_pre_hooks or _global_backward_hooks
   1526                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1527             return forward_call(*args, **kwargs)
   1528 
   1529         try:

[/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py](https://localhost:8080/#) in decorate_context(*args, **kwargs)
    113     def decorate_context(*args, **kwargs):
    114         with ctx_factory():
--> 115             return func(*args, **kwargs)
    116 
    117     return decorate_context

[/usr/local/lib/python3.10/dist-packages/transformers/models/maskformer/modeling_maskformer.py](https://localhost:8080/#) in forward(self, masks_queries_logits, class_queries_logits, mask_labels, class_labels)
    952             # but approximate it in 1 - proba[target class].
    953             # The 1 is a constant that doesn't change the matching, it can be ommitted.
--> 954             cost_class = -pred_probs[:, labels]
    955             # flatten spatial dimension "q h w -> q (h w)"
    956             pred_mask_flat = pred_mask.flatten(1)  # [num_queries, height*width]

IndexError: index 255 is out of bounds for dimension 0 with size 3
```
The same code works without this error when I try it on a different dataset with more classes e.g. `surajbijjahalli/semantic_seg_ATL` is a dataset of different coral species and I have successfully used your same tutorial to train a model on that dataset. However it does not have a pixel value of 255 in the ground truth segmentation maps, which is why i wondered whether a value of 255 is being handled differently, leading to my error. 
Would be grateful if you could point me in the right direction. 

This seems to have to do with the class labels rather than the binary masks. Could you check what `model.config.num_labels` is? The class labels which you provide to the model must be between 0 and `model.config.num_labels - 1`.

`model.config.num_labels` = 2 in my case. The class labels in the ground truth masks are 0 for the background and 255 for the lesion (the class to be segmented). The id2label file is {"0":"BACKGROUND","255": "LESION"}.

If I understand correctly, does this mean that I would have to change the pixel values in the ground truth masks from 255 to 1 ? 

As far as I can see, there would be two ways to do this:
1. Set the mask pixel values to 1 locally i.e. mask[mask==255]=1 before pushing the dataset to hugging face. Change id2label to reflect this as well.
OR
2. Work with the existing dataset and change the mask pixel values after downloading the dataset from hugging face. This will probably mean iterating through each sample in the dataset, changing 255 to 1. 

The first option will not require me to make changes to my code, but I will have to process the dataset differently before pushing to hub. The second option requires me to write routines for handling class label=255. Probably more work but more robust in the long term.
Is there any simpler option I am missing ? 

Thanks again!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.