TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.
Author: hadim
Pull request: no

### System Info

- `transformers` version: 4.36.2
- Platform: macOS-14.2.1-arm64-arm-64bit
- Python version: 3.10.13
- Huggingface_hub version: 0.20.0
- Safetensors version: 0.3.3
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: `mps`
- Using distributed or parallel set-up in script?: NO

### Who can help?

@amyeroberts

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

From https://huggingface.co/facebook/sam-vit-base#usage

```python
import os

# See https://github.com/pytorch/pytorch/issues/77764
os.environ["PYTORCH_ENABLE_MPS_FALLBACK"] = "1"


from PIL import Image
import requests
from transformers import SamModel, SamProcessor

device = "mps"

model = SamModel.from_pretrained("facebook/sam-vit-base")
model = model.to(device)

processor = SamProcessor.from_pretrained("facebook/sam-vit-base")

img_url = "https://huggingface.co/ybelkada/segment-anything/resolve/main/assets/car.png"
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert("RGB")

input_points = [[[450, 600]]]  # THE BUG HAPPENS ONLY WHEN input_points IS NOT None
# input_points = None

inputs = processor(raw_image, input_points=input_points, return_tensors="pt").to(device)
outputs = model(**inputs)
masks = processor.image_processor.post_process_masks(
    outputs.pred_masks.cpu(),
    inputs["original_sizes"].cpu(),
    inputs["reshaped_input_sizes"].cpu(),
)
scores = outputs.iou_scores
```

Error

```
{
	"name": "TypeError",
	"message": "Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead.",
	"stack": "---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[4], line 24
     21 input_points = [[[450, 600]]]
     22 # input_points = None
---> 24 inputs = processor(raw_image, input_points=input_points, return_tensors=\"pt\").to(device)
     25 outputs = model(**inputs)
     26 masks = processor.image_processor.post_process_masks(
     27     outputs.pred_masks.cpu(),
     28     inputs[\"original_sizes\"].cpu(),
     29     inputs[\"reshaped_input_sizes\"].cpu(),
     30 )

File ~/Code/libs/ishisense/.pixi/env/lib/python3.10/site-packages/transformers/feature_extraction_utils.py:231, in BatchFeature.to(self, *args, **kwargs)
    227 for k, v in self.items():
    228     # check if v is a floating point
    229     if torch.is_floating_point(v):
    230         # cast and send to device
--> 231         new_data[k] = v.to(*args, **kwargs)
    232     elif device is not None:
    233         new_data[k] = v.to(device=device)

TypeError: Cannot convert a MPS Tensor to float64 dtype as the MPS framework doesn't support float64. Please use float32 instead."
}
```

### Expected behavior

The script never fails when using `device = "cpu"` and seems to be specific to `device = "mps"`.

Comments:
```diff
+ inputs = processor(raw_image, input_points=input_points, return_tensors="pt").to(torch.float32).to("mps")
- inputs = processor(raw_image, input_points=input_points, return_tensors="pt").to("mps")
```
will fix this. 
I think this is more a mps issue as the conversion works well otherwise. Do you want to open a PR for a fix? 

https://github.com/huggingface/transformers/blob/b1292bca6923cfbc9cb3f70cb55df57e4e17e630/src/transformers/models/sam/processing_sam.py#L123


Using `.to(torch.float32)` makes it work indeed. Thanks.

I'm happy to open a PR but then I am not sure what kind of fix it should contain here since it seems to be more of an `mps` issue and your simple fix makes it work.

Yep let's keep it that way then 😉 

> + inputs = processor(raw_image, input_points=input_points, return_tensors="pt").to(torch.float32).to("mps")
> - inputs = processor(raw_image, input_points=input_points, return_tensors="pt").to("mps")
> will fix this. I think this is more a mps issue as the conversion works well otherwise. Do you want to open a PR for a fix?
> 
> [transformers/src/transformers/models/sam/processing_sam.py](https://github.com/huggingface/transformers/blob/b1292bca6923cfbc9cb3f70cb55df57e4e17e630/src/transformers/models/sam/processing_sam.py#L123)
> 
> Line 123 in [b1292bc](/huggingface/transformers/commit/b1292bca6923cfbc9cb3f70cb55df57e4e17e630)
> 
>  input_points = np.array(input_points)

Checking on this 1.5 years later, and this is still the de facto solution. Thank you!