Possible output bug in the image processing ViT module - causes crash
Author: adaschevici
Pull request: no

### System Info

- `transformers` version: 4.38.2
- Platform: macOS-14.3.1-arm64-arm-64bit
- Python version: 3.11.8
- Huggingface_hub version: 0.21.4
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.2.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes - MPS
- Using distributed or parallel set-up in script?: no


### Who can help?

Tentatively @amyeroberts 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I have used this https://github.com/qdrant/examples/tree/master/qdrant_101_image_data minimal example to get the issue.
The issue I am seeing is that the processing of a PIL image is not being passed into the feature extraction with the expected tensor type.

What i wound up doing is converting it to a tensor at the very last step before it gets passed in transformers/models/vit/image_processing_vit.py:271

A standalone script that highlights the issue is as follows:

```python
from transformers import ViTImageProcessor, ViTModel
from qdrant_client import QdrantClient
from qdrant_client.http import models
from datasets import load_dataset
import numpy as np
import torch

client  = QdrantClient(host="localhost", port=6333)

my_collection = "image_collection"
client.recreate_collection(
    collection_name=my_collection,
    vectors_config=models.VectorParams(size=384, distance=models.Distance.COSINE)
)

my_collection = "image_collection"
client.recreate_collection(
    collection_name=my_collection,
    vectors_config=models.VectorParams(size=384, distance=models.Distance.COSINE)
)

dataset = load_dataset("marmal88/skin_cancer", split="train")

if not torch.backends.mps.is_available():
    if not torch.backends.mps.is_built():
        print("MPS not available because the current PyTorch install was not "
              "built with MPS enabled.")
    else:
        print("MPS not available because the current MacOS version is not 12.3+ "
              "and/or you do not have an MPS-enabled device on this machine.")

else:
    mps_device = torch.device("mps")

processor = ViTImageProcessor.from_pretrained("facebook/dino-vits16")
model = ViTModel.from_pretrained("facebook/dino-vits16").to(mps_device)
image = dataset[8500]["image"]

inputs = processor(images=image, return_tensor="pt").to(mps_device)
inputs["pixel_values"].shape, inputs
```

### Expected behavior

Preprocessing is done successfully and i get this output, to me it seems like a type issue, but i am not entirely sure what the most appropriate solution would be. I have patched my package with the solution posted above.
```sh
(torch.Size([1, 3, 224, 224]),
 {'pixel_values': tensor([[[[-1.3987, -1.4329, -1.4500,  ..., -1.0904, -1.1075, -1.0904],
           [-1.4158, -1.4329, -1.4500,  ..., -1.0904, -1.1075, -1.1075],
           [-1.4329, -1.4500, -1.4500,  ..., -1.1247, -1.1075, -1.1075],
           ...,
           [-1.1075, -1.1075, -1.0904,  ..., -1.4843, -1.5014, -1.5357],
           [-1.1075, -1.1075, -1.0904,  ..., -1.4843, -1.5185, -1.5528],
           [-1.1247, -1.1247, -1.0904,  ..., -1.4843, -1.5357, -1.5528]],
 
          [[-1.7381, -1.7381, -1.7556,  ..., -1.5980, -1.6155, -1.6331],
           [-1.7381, -1.7381, -1.7556,  ..., -1.6155, -1.6331, -1.6506],
           [-1.7381, -1.7381, -1.7556,  ..., -1.5980, -1.6155, -1.6155],
           ...,
           [-1.5630, -1.5630, -1.5630,  ..., -1.7731, -1.7906, -1.7906],
           [-1.5630, -1.5630, -1.5630,  ..., -1.7906, -1.7906, -1.8081],
           [-1.5805, -1.5805, -1.5630,  ..., -1.7906, -1.8081, -1.8081]],
 
          [[-1.3513, -1.3687, -1.3861,  ..., -1.2119, -1.2119, -1.2293],
           [-1.3687, -1.3687, -1.3861,  ..., -1.2119, -1.2293, -1.2467],
           [-1.3687, -1.3861, -1.4036,  ..., -1.2119, -1.2119, -1.2293],
           ...,
           [-1.1770, -1.1770, -1.1596,  ..., -1.4559, -1.4907, -1.5081],
           [-1.1596, -1.1770, -1.1596,  ..., -1.4733, -1.4907, -1.5081],
           [-1.1596, -1.1770, -1.1770,  ..., -1.4733, -1.4733, -1.5081]]]],
        device='mps:0')})
```

Comments:
Hi @adaschevici, thanks for raising an issue! 

> What i wound up doing is converting it to a tensor at the very last step before it gets passed in transformers/models/vit/image_processing_vit.py:271

Could you clarify this? From the code example, it looks like the outputs of the image processor are being sent to the mps device after processing in this line:

```py
inputs = processor(images=image, return_tensors="pt").to(mps_device)
```

rather converting before passing to the image processor. 

Moving the outputs to the device after processing is the correct way to do this. 

Note: there's also a typo in the example, it should be `return_tensors` rather than `return_tensor`. 

oh, wow. It was the typo. apologies. I was getting this trace and it threw me off

```python
226 # We cast only floating point tensors to avoid issues with tokenizers casting `LongTensor` to `FloatTensor`
    227 for k, v in self.items():
    228     # check if v is a floating point
--> 229     if torch.is_floating_point(v):
    230         # cast and send to device
    231         new_data[k] = v.to(*args, **kwargs)
    232     elif device is not None:

TypeError: is_floating_point(): argument 'input' (position 1) must be Tensor, not list
```
Closing this. Thanks for pointing it out.