post_process_instance_segmentation does not resize outputs?
Author: nickponline
Pull request: no

### System Info

```
Python 3.8.8 (default, Apr 13 2021, 15:08:03) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
import transformers
transformers.__version__
'4.25.1'
```

### Who can help?
```
results = processor.post_process_instance_segmentation(outputs, target_sizes=[(1000, 1000), (1000, 1000)] )
results[0]['segmentation'].cpu().numpy().shape
(128, 128)
```

Shouldn't the output be ```(1000, 1000)```

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```
results = processor.post_process_instance_segmentation(outputs, target_sizes=[(1000, 1000), (1000, 1000)] )
results[0]['segmentation'].cpu().numpy().shape
(128, 128)
```

Shouldn't the output be `(1000, 1000)`

### Expected behavior

Output masks should be same size as `target_sizes`

Comments:
cc @alaradirik 

@alaradirik @NielsRogge what's weird is `post_process_semantic_segmentation` works as expects but not `post_process_instance_segmentation`

```
results = processor.post_process_semantic_segmentation(outputs, target_sizes=[(1000, 1000), (1000, 1000)] )
results[0]['segmentation'].cpu().numpy().shape
(1000, 1000)
```


```
results = processor.post_process_instance_segmentation(outputs, target_sizes=[(1000, 1000), (1000, 1000)] )
results[0]['segmentation'].cpu().numpy().shape
(128, 128)
```

Hi @nickponline!

Are you using MaskFormer or Mask2Former? We are aware of the issue and it should be fixed with the latest release. Could you try upgrading to transformers 4.26.0?

MaskFormer

On Fri, Jan 27, 2023 at 12:08 AM Alara Dirik ***@***.***>
wrote:

> Hi @nickponline <https://github.com/nickponline>!
>
> Are you using MaskFormer or Mask2Former? We are aware of the issue and it
> should be fixed with the latest release. Could you try upgrading to
> transformers 4.26.0?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/21313#issuecomment-1406151217>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAEQCR2MY7H2MV5ED6QIEJTWUN66PANCNFSM6AAAAAAUHDZ5ZI>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>


@alaradirik @NielsRogge when I use 4.26.0 the resizing works and I get masks back of the `target_size`, but I do notice that the fidelity of the semantic segmentation results are better than the instance segmentation ones. Is something different in how the masks are upsampled? Example below and both masks are 1000x1000

 instance segmentation  (notice the low resolution) 

![instance](https://user-images.githubusercontent.com/590151/215220060-b91b5d3e-dc06-4a58-969b-77a38e8b52e3.png)

 semantic semantic segmentation (right). 

![semantic](https://user-images.githubusercontent.com/590151/215220075-b86e18d9-0a4a-46e6-8686-520e54fbe28b.png)





@alaradirik these results are with Mask2Former ^

@nickponline that is a good point. 

To answer your question, Mask2Former outputs mask logits of shape (96, 96) for efficiency purposes. The `post_process_semantic_segmentation` method directly interpolates the mask logits to the target size, whereas the `post_process_instance_segmentation` method first interpolates the mask logits to the preprocessed image size (384, 384), computes the final score of each binary mask proposal by multiplying the mask proposal score with the class score and resizes the final instance segmentation map (discrete instance id values instead of continuous logit values) . 

The `post_process_instance_segmentation` method yields the same results as the original Mask2Former post-processing. However, you can smoothen the results by cloning the repo, editing line 990 of the [image_processing_mask2former.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mask2former/image_processing_mask2former.py) file such that the mask logits are interpolated to the target size (instead of (384, 384)) and building the library locally.


@alaradirik I also noticed the hard-coded resizing to (384, 384) in `post_process_instance_segmentation`. What if the user chooses to resize the input images to a different input size, e.g., (480, 640)? Shouldn't the post-processing adapt to the actual input size?

@Callidior you can pass `target_sizes` to the `post_process_instance_segmentation` method, which is a list containing the desired (height, width) as tuples

@NielsRogge As far as I understood, resizing to `target_sizes` happens after the hard-coded resize to 384x384. It does not replace it.
But what if I resize the input images, for example, to a non-quadratic size of 640x960. The post-processing would first resize the segmentation maps to 384x384 and then to 640x960. This would loose much more spatial precision along one dimension than the other.

Yes but as @alaradirik explains, this is to comply to the original implementation, which always first interpolates to (384x384). So as she suggest, if you really want to interpolate directly to the desired size, feel free to fork and edit [this line](https://github.com/huggingface/transformers/blob/2ea1ef909016484bee9d60c05582031464490f77/src/transformers/models/mask2former/image_processing_mask2former.py#L990) 

Hi @Callidior, as @NielsRogge pointed out, we follow the original implementation in order to make it easier for users to benchmark official checkpoints or their fine-tuned model against other models. 

In order to make changes, you can git clone the repo, change the relevant line and install the library locally with `pip install -e ".[dev]"`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.