ImageSegmentation pipeline error when calling `.to(dtype)` in preprocessor of OneFormer models
Author: philkuz
Pull request: no
Labels: name: bug

### System Info

> `transformers-cli env`
```
- `transformers` version: 4.44.2
- Platform: Linux-6.5.0-21-generic-x86_64-with-glibc2.35
- Python version: 3.11.9
- Huggingface_hub version: 0.24.6
- Safetensors version: 0.4.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.3.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: No
- GPU type: NVIDIA GeForce RTX 4090
```


### Who can help?

cc: @Rocketknight1 @amyeroberts @qubvel 

### Information

- [X] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I try to run the following script:
```
import requests
from PIL import Image
from transformers import pipeline

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)


pipe = pipeline("image-segmentation", model="shi-labs/oneformer_cityscapes_swin_large")
pipe(image, subtask="semantic")
```
But get the following issue:
```
/home/philkuz/.pyenv/versions/gml311/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be depracted in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
  warnings.warn(
/home/philkuz/.pyenv/versions/gml311/lib/python3.11/site-packages/transformers/utils/deprecation.py:165: UserWarning: The following named arguments are not valid for `OneFormerImageProcessor.__init__` and were ignored: '_max_size'
  return func(*args, **kwargs)
Hardware accelerator e.g. GPU is available in the environment, but no `device` argument is passed to the `Pipeline` object. Model will be on CPU.
Traceback (most recent call last):
  File "/home/philkuz/gimlet/fail.py", line 11, in <module>
    pipe(image, subtask="semantic")
  File "/home/philkuz/.pyenv/versions/gml311/lib/python3.11/site-packages/transformers/pipelines/image_segmentation.py", line 139, in __call__
    return super().__call__(images, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/philkuz/.pyenv/versions/gml311/lib/python3.11/site-packages/transformers/pipelines/base.py", line 1257, in __call__
    return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/philkuz/.pyenv/versions/gml311/lib/python3.11/site-packages/transformers/pipelines/base.py", line 1263, in run_single
    model_inputs = self.preprocess(inputs, **preprocess_params)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/philkuz/.pyenv/versions/gml311/lib/python3.11/site-packages/transformers/pipelines/image_segmentation.py", line 151, in preprocess
    inputs = inputs.to(self.torch_dtype)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/philkuz/.pyenv/versions/gml311/lib/python3.11/site-packages/transformers/feature_extraction_utils.py", line 230, in to
    if torch.is_floating_point(v):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: is_floating_point(): argument 'input' (position 1) must be Tensor, not list
```

I'm pretty sure the main issue is:
```
  File "/home/philkuz/.pyenv/versions/gml311/lib/python3.11/site-packages/transformers/pipelines/image_segmentation.py", line 151, in preprocess
    inputs = inputs.to(self.torch_dtype)
```

which was introduced in https://github.com/huggingface/transformers/pull/31342/files as a way to support FP16 across all pipelines, however the OneFormer path was probably not tested? Especially since it's a special path through the code.



### Expected behavior

I expect to be able to run the OneFormer model through ImageSegmentation, especially given there is a clear special case for it in the pipeline API.

Happy to help fix it

Comments:
Actually looks like this issue was addressed in https://github.com/huggingface/transformers/pull/33918! I forked the repo and found that I could not longer repro the bug. I'll wait for the next release!

Good news @philkuz! Next release will likely be next week around the 23rd.

Thank you @LysandreJik. Eagerly awaiting it and getting by with my fork