AssertionError when converting openai clip's weight to hf with higher transformers version 
Author: Yufang-Liu
Pull request: no
Labels: name: Multimodal

### System Info

transformers version: 4.39.0
python version: 3.10
torch version(gpu): 2.0.0

### Who can help?

@amyeroberts 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

with transformers==4.39.0, run with `python .\convert_clip_original_pytorch_to_hf.py --checkpoint_path ViT-B/32 --pytorch_dump_folder_path ./`, the error is: 

```Traceback (most recent call last):
  File "C:\Users\Yufang Liu\Downloads\convert_clip_original_pytorch_to_hf.py", line 148, in <module>
    convert_clip_checkpoint(args.checkpoint_path, args.pytorch_dump_folder_path, args.config_path)
  File "D:\anaconda3\envs\pt\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\Yufang Liu\Downloads\convert_clip_original_pytorch_to_hf.py", line 135, in convert_clip_checkpoint
    assert torch.allclose(hf_logits_per_image, pt_logits_per_image, atol=1e-3)
AssertionError```
```

with transformers==4.30.0, the error is gone. I also tried 4.32.0, same error.

### Expected behavior

A hf's clip weight should be generated from original openai's by running this script.

Comments:
cc @NielsRogge as well

Hi @Yufang-Liu, thanks for reporting! 

This broke with #24777 - @ydshieh could you take a look? 

Reopen as the conversion is still not working (with other errors) yet

@Narsil Could you take a look for the following error(pull the main first)

My env:
```
Name: safetensors
Version: 0.4.3
```

Code snippet
```bash
cd src/transformers/models/clip
pip install ftfy regex tqdm && pip install git+https://github.com/openai/CLIP.git
python3 convert_clip_original_pytorch_to_hf.py --checkpoint_path ViT-B/32 --pytorch_dump_folder_path ./
```

gives

```bash
Traceback (most recent call last):
  File "convert_clip_original_pytorch_to_hf.py", line 156, in <module>
    convert_clip_checkpoint(args.checkpoint_path, args.pytorch_dump_folder_path, args.config_path)
  File "/usr/local/lib/python3.8/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "convert_clip_original_pytorch_to_hf.py", line 146, in convert_clip_checkpoint
    hf_model.save_pretrained(pytorch_dump_folder_path)
  File "/transformers/src/transformers/modeling_utils.py", line 2585, in save_pretrained
    safe_save_file(shard, os.path.join(save_directory, shard_file), metadata={"format": "pt"})
  File "/usr/local/lib/python3.8/dist-packages/safetensors/torch.py", line 284, in save_file
    serialize_file(_flatten(tensors), filename, metadata=metadata)
  File "/usr/local/lib/python3.8/dist-packages/safetensors/torch.py", line 488, in _flatten
    return {
  File "/usr/local/lib/python3.8/dist-packages/safetensors/torch.py", line 492, in <dictcomp>
    "data": _tobytes(v, k),
  File "/usr/local/lib/python3.8/dist-packages/safetensors/torch.py", line 406, in _tobytes
    raise ValueError(
ValueError: You are trying to save a non contiguous tensor: `visual_projection.weight` which is not allowed. It either means you are trying to save tensors which are reference of each other in which case it's recommended to save only the full tensors, and reslice at load time, or simply call `.contiguous()` on your tensor to pack it before saving.
```

@Yufang-Liu

Fixed now :-)