RuntimeError occurs when using CLIPProcessor and CLIPModel on aarch64
Author: shunkominato
Pull request: no

### System Info

Hi thank you for the great library

Running on docker


The machine environment is as follows
```
aarch64
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
```

host machine
```
Apple M2 tip
Ventura 13.0
```

transformers-cli env environment is as follows
```
- `transformers` version: 4.35.2
- Platform: Linux-6.4.16-linuxkit-aarch64-with-glibc2.31
- Python version: 3.9.2
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```
pip3 list environment is as follows
```
Package            Version
------------------ ----------
certifi            2023.11.17
charset-normalizer 3.3.2
filelock           3.13.1
fsspec             2023.10.0
huggingface-hub    0.19.4
idna               3.6
Jinja2             3.1.2
MarkupSafe         2.1.3
mpmath             1.3.0
networkx           3.2.1
numpy              1.26.2
packaging          23.2
Pillow             10.1.0
pip                20.3.4
PyYAML             6.0.1
regex              2023.10.3
requests           2.31.0
safetensors        0.4.1
setuptools         52.0.0
sympy              1.12
tokenizers         0.15.0
torch              2.1.1
torchvision        0.16.1
tqdm               4.66.1
transformers       4.35.2
typing-extensions  4.8.0
urllib3            2.1.0
wheel              0.34.2
```



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The code executed is below
```
import torch
from PIL import Image
from transformers import CLIPProcessor, CLIPModel

emotions = ["喜び", "悲しみ", "怒り", "驚き", "恐怖", "嫌悪"]
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")

image = Image.open("1.jpg")
inputs = processor(text=emotions, images=image, return_tensors="pt", padding=True)
outputs = model(**inputs)

logits_per_image = outputs.logits_per_image

probs = logits_per_image.softmax(dim=1)

for i, emotion in enumerate(emotions):
    print(f"{emotion}: {probs[0][i].item()}")
```
An error occurs in the following places
`outputs = model(**inputs)`

### Expected behavior

When you run the code in the above environment
```
Traceback (most recent call last):
  File "/app/lib/ai/image_ai.py", line 28, in <module>
    outputs = model(**inputs)
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/transformers/models/clip/modeling_clip.py", line 1130, in forward
    logits_per_text = torch.matmul(text_embeds, image_embeds.t()) * logit_scale
RuntimeError: could not create a primitive descriptor for a matmul primitive
```



We also confirmed that it ran without any problems when executed on x86_64.

Sorry if there is a similar issue


Comments:
Hey! Thanks for reporting, never had an issue with my macs and running in native code, would recommend you to just run on the native os. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Running inside of a container (aarch64 on MacOS arm64) is a useful feature.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.