handling edge case for handling void segmentation maps in the image processor
Author: JINO-ROHIT
Pull request: yes

# This PR handles the edge case when a void segmentation maps is passed to the image processor

Fixes #30064 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts @NielsRogge 


Comments:
Hi @JINO-ROHIT, thanks a lot for the PR!

There is a similar code for MaskFormer and OneFormer models, please make sure to make changes for them too. You could run `make fix-copies` for that. 
It would be also great to have some test that checks this corner case working fine now 🙂 

Hi @qubvel sure im on it!
About the tests part, Ive never written tests for this library before. Do you mind guiding me a little bit as to how to write for this?
Thanks so much!

For each model, there are `image_processor` tests in `tests/models/<model>/test_image_processing_<model>.py`. You can look at the tests there and get inspiration on how to write a new one 🙂 

https://github.com/huggingface/transformers/blob/main/tests/models/maskformer/test_image_processing_maskformer.py

[Here](https://huggingface.co/docs/transformers/testing#writing-tests) is a general guide on how to write tests.


thanks @qubvel ill work on the tests, meanwhile im not sure why the other test are failing, they seem unrelated and some hardware issue i think

@qubvel i need some guidance here if you dont mind - 
so what i observe is when Mask2FormerImageProcessor is called -
there is a preprocess function for image and masks and then it is encoded, so for the unit test should I import all those class methods and write the unit test?

For the image processor, there is a test for `__call__` function, you could probably extend it for this corner case
https://github.com/huggingface/transformers/blob/8c12690cecbb97e187861e386f7a0ac790e4236c/tests/models/maskformer/test_image_processing_maskformer.py#L222

You can also add separate tests for the modified `convert_segmentation_map_to_binary_masks` function similar to 
https://github.com/huggingface/transformers/blob/8c12690cecbb97e187861e386f7a0ac790e4236c/tests/models/maskformer/test_image_processing_maskformer.py#L406

Hope that helps, and let me know if you will have any questions 🙂 

right thanks! 

@qubvel is this better?

Hi @JINO-ROHIT 

Thanks for writing a test case! I would suggest adding different scenarios to check behavior:
 - multiple labels
 - multiple labels + ignore_index
 - two labels
 - two labels + ignore_index
 - one label (already covered)
 - one label != ignore_index
 - one label == ignore_index

The most interesting from my point of view is `one label == ignore_index` (for example, we have only background and want to ignore it). Should we return a mask full of `1` or full of `0`? How would the model behave in both cases? 

Let's suppose we have the label `255` for a background class and want to ignore it.
1) In case the image has class `1` and background (`255`). The function will return a binary mask for the `1` class label and will **not** return a mask for the background (`255`) class label.
2) In case the image has only a background (`255` class label). The function will return a binary mask for `255` class label full of ones.

There is a kind of inconsistency in the behavior of the function, what do you think about it? 

Let me know if I missed something, and thanks once again for proposing a PR to fix this issue 🙂 

@qubvel thanks for such a detailed write up, it makes sense and valid points tbh, ill take a lil more time over the weekend and cover these cases extensively!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.