Image Processor fails to process void segmentation maps
Author: antopost
Pull request: no
Labels: name: Good Second Issue

name: Feature request

name: Vision

### System Info

- `transformers` version: 4.34.0
- Platform: Linux-5.15.0-89-generic-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.21.4
- Safetensors version: 0.4.0
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import Mask2FormerImageProcessor
import numpy as np


ignore_index = 255
image_processor = Mask2FormerImageProcessor(ignore_index=ignore_index,
                                                         do_resize=False,
                                                         do_rescale=False,
                                                         do_normalize=False)

image_norm = np.random.rand(4, 4, 3)

# Create void mask (all pixels have ignore_index)
semantic_mask = np.ones(image_norm.shape[:2], dtype=np.uint8)*255

semantic_mask = semantic_mask.astype(np.uint8)
print(semantic_mask)

inputs = image_processor(
    image_norm,
    segmentation_maps=semantic_mask,
    return_tensors='pt',
)

print(inputs)
```


===========================================================

> [[255 255 255 255]
>  [255 255 255 255]
>  [255 255 255 255]
>  [255 255 255 255]]
> Traceback (most recent call last):
>   File "/home/anba/catkin_ws/src/tas_dev/dev/anba/Mask2Former/test.py", line 21, in <module>
>     inputs = image_processor(
>   File "/home/anba/anaconda3/envs/SAM/lib/python3.10/site-packages/transformers/models/mask2former/image_processing_mask2former.py", line 566, in __call__
>     return self.preprocess(images, segmentation_maps=segmentation_maps, **kwargs)
>   File "/home/anba/anaconda3/envs/SAM/lib/python3.10/site-packages/transformers/models/mask2former/image_processing_mask2former.py", line 764, in preprocess
>     encoded_inputs = self.encode_inputs(
>   File "/home/anba/anaconda3/envs/SAM/lib/python3.10/site-packages/transformers/models/mask2former/image_processing_mask2former.py", line 943, in encode_inputs
>     masks, classes = self.convert_segmentation_map_to_binary_masks(
>   File "/home/anba/anaconda3/envs/SAM/lib/python3.10/site-packages/transformers/models/mask2former/image_processing_mask2former.py", line 558, in convert_segmentation_map_to_binary_masks
>     return convert_segmentation_map_to_binary_masks(
>   File "/home/anba/anaconda3/envs/SAM/lib/python3.10/site-packages/transformers/models/mask2former/image_processing_mask2former.py", line 284, in convert_segmentation_map_to_binary_masks
>     binary_masks = np.stack(binary_masks, axis=0)  # (num_labels, height, width)
>   File "<__array_function__ internals>", line 180, in stack
>   File "/home/anba/anaconda3/envs/SAM/lib/python3.10/site-packages/numpy/core/shape_base.py", line 422, in stack
>     raise ValueError('need at least one array to stack')
> ValueError: need at least one array to stack
> 
> Process finished with exit code 1

### Expected behavior

If this is intended that void masks should never be passed, then the result is fine.

However, when training segmentation models, shouldn't it be possible to include images with only background/void class?

Comments:
cc @amyeroberts and @NielsRogge 

This looks like an edge case which isn't supported yet. Would require a PR to extend it for this use case

can i take this up? @NielsRogge 

@JINO-ROHIT Sure! 

Any work around for this issue ?  I am getting this error often for my use case. 

@shriyashchougule The other PR closed due to inactivity - feel free to open open if you'd like to tackle the issue! 

Hey @amyeroberts, I'd like to tackle this issue if it's still not solved.

@Lynsoo it's been a while since this issue was opened - cc @qubvel @nielsrogge to check it hasn't been resolved already!

There was an issue resolved with void output masks, but as far as I understand, this one is related to input masks. @Lynsoo, feel free to double-check if the issue still exists and fix it 🤗

Ok I'll check and work on that. Thank you @Rocketknight1 @qubvel. 

Hi, I noticed this PR is still open and hasn’t been finalized yet. I’m new to open source, but I’d love to give it a try and do my best to help finish it. Would it be okay if I work on this?

<img width="1503" height="833" alt="Image" src="https://github.com/user-attachments/assets/a46bd519-b4fa-4f39-a587-99c51581a994" />
I've python==3.10.18 and transformers==4.55.0 and it seems working fine. Please check and if it's already done then close this PR please.

Hi, 

Is this issue still open? I see some PRs mentioned in the comments, could someone tell me what the status is?

cc @NielsRogge @molbap not sure if we should close this one!

Verified on current `main` (`4347de89`, v5.10.0.dev0): the original repro and several variants — all-`ignore_index` maps, all-background maps with `do_reduce_labels=True`, instance-id mappings, and batches mixing void and non-void maps — now work for both the PIL and torchvision image processors of MaskFormer and Mask2Former, returning empty `(0, H, W)` `mask_labels` and empty `class_labels` as expected. The fix appears to have landed during the v5 processor rewrites (all eight `convert_segmentation_map_to_binary_masks` variants across MaskFormer/Mask2Former/OneFormer/EOMT now guard the empty-mask case), but I could not find a regression test covering void maps in any of those models' test suites.

I would like to close the loop with a tests-only PR adding `test_call_with_void_segmentation_maps` to the MaskFormer and Mask2Former image-processing tests (happy to extend to OneFormer/EOMT if useful). The branch is ready: https://github.com/JohnnyWilson-Portfolio/transformers/tree/add-void-segmentation-map-regression-tests — the new tests pass on `main`, and they fail with the original `ValueError: need at least one array to stack` if the empty-mask guards are removed, so they act as a real regression tripwire.

Disclosure per the agentic-contributions guidelines: the investigation and test drafting were AI-assisted (Claude Code); all test runs were executed locally (macOS, torch 2.12). Unlike the stale #38532, this approach keeps void samples in the batch and touches no library code. Should I open the PR?

How to ues