On branch fix-void-segment-mask-input [WIP]
Author: nam-drun
Pull request: yes

Changes to be committed:
	modified:   src/transformers/models/mask2former/image_processing_mask2former.py

Fix [#30064](https://github.com/huggingface/transformers/issues/30064)
This differ from previous PR fix [#30335](https://github.com/huggingface/transformers/pull/30335)? This fix tackles an edge case where an image's segment map only has background pixel (255 class label) whereas the previous PR fix doesn't.

### How is this PR fix intended to works?
Let's input a batch into `class Mask2FormerImageProcessor`, where `images = [img_1, img_2, img_3,..., img_B]` and its corresponding `segmentation maps = [map_1, map_2,  map_3,... , map_B]`. Suppose `map_1` and `map_2` are void maps, then `def filter_image_with_void_segmentation_map` will pop void maps and its corresponding images, returning only `images = [img_3,..., img_B]` and `segmentation_maps = [map_3,..., map_B]`

I believe several testings will be desired, please advice.

### Before submitting
This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
[documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
[here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

### Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@amyeroberts @NielsRogge @Lynsoo 

### Things to work on before this PR works, I think
- [ ] Implement `AdaptiveBatchSizeTrainer` to [continue model's training despite reduced batch size](https://github.com/huggingface/transformers/pull/38532#issuecomment-2937069511)
- [ ] Add a specific condition to only filter void map during validation. That's my interpretation of OP's question in [#30064](https://github.com/huggingface/transformers/issues/30064): "when training segmentation models, shouldn't it be possible to include images with only background/void class?" But this question is never addressed so idk whether you guys want void class when training model or no?


Comments:
cc @qubvel

Let's suppose the model expects batch size `B`, but after filtering out 10 void maps, it only receives `B-10` batch size. Does HuggingFace already have a solution for this situation?

Technically this is a solution if we consider `AdaptiveBatchSizeTrainer` since it supports dynamically changing batch sizes.
@Skimmable-Code-pls did you actually test it with a void-injected batch ?

> Technically this is a solution if we consider `AdaptiveBatchSizeTrainer` since it supports dynamically changing batch sizes. @Skimmable-Code-pls did you actually test it with a void-injected batch ?

Not yet, I think my code wouldn't work until `AdaptiveBatchSizeTrainer` is implemented somehow. Where do you advice to implement or call `AdaptiveBatchSizeTrainer`?

@Skimmable-Code-pls  Well you’ll need to define a custom batch scheduler and then pass it as an argument to AdaptiveBatchSizeTrainer and then use AdaptiveBatchSizeTrainer instead of the regular Trainer in the training loop. If you need any assistance feel free to ask. 

> use AdaptiveBatchSizeTrainer instead of the regular Trainer in the training loop

@Lynsoo here is my test script [void_batch_test.py](https://github.com/Skimmable-Code-pls/transformers/blob/fix-void-segment-mask-input/void_batch_test.py). I take inspirations from Mask2Former HF doc and [ifiaposto/hf_trainer_with_online_batch_size repo](https://github.com/ifiaposto/hf_trainer_with_online_batch_size/tree/main)

I leave `training_args` up to you to decide. If I may ask, can you run my test script for me? I don't have GPU.

@Lynsoo My apology, I will test `void_batch_test.py` next weekends, it's just this week I'm too cramped by work

@Skimmable-Code-pls It’s okay. I’m just busy these days and probably won’t be available until around mid July but I’ll try to check the test file today or tomorrow.