Can i convert open-clip trained models (.pt) using code “src/transformers/models/clip/convert_clip_original_pytorch_to_hf.py” ？
Author: jzssz
Pull request: no
Labels: name: New model

### Model description

openclip：https://github.com/mlfoundations/open_clip

i use openclip  to train model and get "epoch_400.pt". 
**and i want to convert this "epoch_400.pt" to hf, so i run:**
python src/transformers/models/clip/convert_clip_original_pytorch_to_hf.py     --pytorch_dump_folder_path "./openclip_syf_hf"     --checkpoint_path "/openclip_output/2023_12_07-15_24_24-model_ViT-B-32-lr_0.0005-b_256-j_8-p_amp/checkpoints/epoch_400.pt"     --config_path "/open_clip-main/src/open_clip/model_configs/ViT-B-32.json"

**but get bug:**

Traceback (most recent call last):
  File "/home/anaconda3/envs/transformer/lib/python3.8/site-packages/clip/clip.py", line 130, in load
    model = torch.jit.load(opened_file, map_location=device if jit else "cpu").eval()
  File "/home/anaconda3/envs/transformer/lib/python3.8/site-packages/torch/jit/_serialization.py", line 164, in load
    cpp_module = torch._C.import_ir_module_from_buffer(
RuntimeError: PytorchStreamReader failed locating file constants.pkl: file not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/transformers/models/clip/convert_clip_original_pytorch_to_hf.py", line 150, in <module>
    convert_clip_checkpoint(args.checkpoint_path, args.pytorch_dump_folder_path, args.config_path)
  File "/home/anaconda3/envs/transformer/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "src/transformers/models/clip/convert_clip_original_pytorch_to_hf.py", line 120, in convert_clip_checkpoint
    pt_model, _ = load(checkpoint_path, device="cpu", jit=False)
  File "/home/anaconda3/envs/transformer/lib/python3.8/site-packages/clip/clip.py", line 137, in load
    state_dict = torch.load(opened_file, map_location="cpu")
  File "/home/anaconda3/envs/transformer/lib/python3.8/site-packages/torch/serialization.py", line 795, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/home/anaconda3/envs/transformer/lib/python3.8/site-packages/torch/serialization.py", line 1002, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
EOFError: Ran out of input

**so i am wondering if i can convert open-clip trained models (.pt) using code “src/transformers/models/clip/convert_clip_original_pytorch_to_hf.py” ？**

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_

Comments:
Hi @jzssz, thanks for raising an issue! 

Yes, it should be possible to convert a CLIP checkpoint to its HF equivalent using the script. Without having access to your checkpoint, it's not possible to reproduce and debug the issue on our side. 

From the error, it looks like there's a problem in the serialization of the original checkpoint itself: the error is being raised in the pickle library when trying to load the state dict. You'll need to make sure you can load the original checkpoint in pytorch to ensure that the script can be used. 

Hi,

This script can be used to convert OpenCLP models to HF: https://gist.github.com/rwightman/c79fd0241ed3c860e898114931c07990. Would perhaps be great to add it to the "clip" folder as a general utility script

> https://gist.github.com/rwightman/c79fd0241ed3c860e898114931c07990

Do you have a way yo convert [clip-italian](https://github.com/clip-italian/clip-italian) to open_clip?