Convert MetaCLIP checkpoints
Author: NielsRogge
Pull request: no
Labels: name: Good First Issue

### Feature request

Would be great to port the MetaCLIP checkpoints released by Meta to the HF format.

Link: https://github.com/facebookresearch/MetaCLIP

### Motivation

MetaCLIP claims to reproduce the pipeline that OpenAI used when creating CLIP, outperforming the original models.

### Your contribution

I could do this myself but would be great if someone else can take this up

Comments:
@NielsRogge Hi! I will do it. 

Hi @NielsRogge, I am new to all of this and not sure what HF format is. Could you please help me with some resources to learn about it a little more.


@plon-Susk7  / @Natyren , The translation of these weights should more or less follow a similar pattern to the conversion script here: https://github.com/huggingface/transformers/blob/main/src/transformers/models/clip/convert_clip_original_pytorch_to_hf.py

If you haven't seen it already, I'd also suggest looking through [this doc](https://huggingface.co/docs/transformers/add_new_model) as it covers in detail all the steps for adding a model to transformers and will give you a better idea of what hf format means.


Yes @shauray8 although that script was used to convert the OpenAI CLIP model to the HF format.

The MetaCLIP checkpoints are in the OpenCLIP format, for which you can use this script: https://gist.github.com/rwightman/c79fd0241ed3c860e898114931c07990

> Yes @shauray8 although that script was used to convert the OpenAI CLIP model to the HF format.
> 
> The MetaCLIP checkpoints are in the OpenCLIP format, for which you can use this script: https://gist.github.com/rwightman/c79fd0241ed3c860e898114931c07990

I tried to run this code but couldn't pass the allclose assert here even with other open_clip models https://gist.github.com/rwightman/c79fd0241ed3c860e898114931c07990#file-convert_open_clip_to_hf-py-L235
Would like to see if it works for other folks

@TonyZhanghm the MetaCLIP checkpoints use a different activation function as pointed out in their README. You need to adjust it accordingly when creating the `CLIPConfig`

@NielsRogge Hey, Can I take this issue?

@NielsRogge @shauray8 Hi!
I have converted the weights and am currently in the loading stage (by the way, could you please advise on the best way to do this into Meta AI hf hub?). Regarding the initial scripts, I want to point out that the model incorrectly processes torch.arange(0, 77) tokens. To check this, I tokenized real text using open_clip.tokenize(), and in this case, everything is working fine.
Here is the new script.
https://gist.github.com/Natyren/c7d7889095e8e06df76e8316c2fcf89e

@NielsRogge I've uploaded the weights (here https://huggingface.co/GeorgeBredis/MetaCLIP_b32_400m), could you please tell me if I can upload them to the original Meta repository? As far as I understand, I may need additional permissions for this.
I've also converted the weights of the remaining models and am waiting for approval to upload them.

Hi @Natyren, thanks for converting them and updating the script accordingly!

Feel free to place them under your username, we'll transfer them to the Meta organization. Also, a minor thing, could you use the format `metaclip-b32-400m` for instance regarding naming of the checkpoints (just lowercase please)

@NielsRogge Thank you for your response. I will soon upload it with the correct names

@NielsRogge  I've uploaded all the models, you can find them there (https://huggingface.co/GeorgeBredis), but there are no preprocessing and tokenizer configs in these versions, but everything should work fine with them. If something doesn't work smoothly, feel free to ping and write me! 

Awesome, great work! I guess we can also add the `CLIPImageProcessor` and `CLIPTokenizer` files. Usually I do the following to make sure the inputs are prepared in the same way:
```
import torch
from PIL import Image
import open_clip
from transformers import CLIPImageProcessor, CLIPTokenizer

model, _, preprocess = open_clip.create_model_and_transforms('ViT-B-32-quickgelu', pretrained='metaclip/b32_400m.pt')

image = Image.open("CLIP.png")
original_pixel_values = preprocess(image).unsqueeze(0)
text = ["a diagram", "a dog", "a cat"]
original_input_ids = open_clip.tokenize(text)

# verify pixel_values
image_processor = CLIPImageProcessor()
pixel_values = image_processor(image, return_tensors="pt").pixel_values
assert torch.allclose(pixel_values, original_pixel_values)

# verify input_ids
tokenizer = CLIPTokenizer.from_pretrained("openai/clip-vit-base-patch32")
input_ids = image_processor(text, return_tensors="pt").input_ids
assert torch.allclose(input_ids, original_input_ids)
```

If those pass, then we can create a CLIPProcessor which wraps both, and then push those to the hub.
```
from transformers import CLIPProcessor

processor = CLIPProcessor(image_processor=image_processor, tokenizer=tokenizer)
processor.push_to_hub("...")
```

@NielsRogge Uploaded, preprocessed in the repositories (https://huggingface.co/GeorgeBredis). Here is the code for testing:
https://gist.github.com/Natyren/47e564357cac95bf923b7c65781492c2
After testing the model's functionality in the WebUI, everything is fine.
If something go wrong, feel free to write me

Awesome work 👏 will transfer all checkpoints to the `facebook` organization

There you have it: 7 checkpoints with added model cards: https://huggingface.co/models?other=metaclip.

Thanks a lot 🙏 looks like they're still training a giant-sized version on the 2.5 billion samples.

Will be interesting to see whether these models are as good (or better) than the ones OpenAI released.

Good job, thank you. As soon as they add it, I will upload it too. Yes, I agree, it will be interesting to see what they achieve in terms of metrics.

@Natyren some people reported to me that some MetaCLIP models on the hub are using "gelu" instead of "quickgelu", did you verify all conversions? e.g. https://huggingface.co/facebook/metaclip-h14-fullcc2.5b/blob/main/config.json#L10

@NielsRogge Hello! Yes, you are correct. Upon taking a closer look at the Meta results, I see that they have decided to follow the original architecture of CLIP. I didn't pay special attention to this as some of the models with GELU activation were showing similar results during transfer. There are two possible solutions here. First, we can simply replace the activation with quick-gelu in the repository. Second, I will recreate the models with the correct architecture (in theory, the results should be the same except for the config). I apologize for this oversight.

Sure, let me locally test the difference with the config change and a complete model re-save (just to be safe). If everything turns out to be identical, we can replace the configs. If not, I will upload a new version.


I've just checked, and the results are significantly different (though I haven't figured out why yet). I'll make the necessary corrections and upload new weights within the next day. I apologize for any inconvenience this may have caused.

@NielsRogge I've uploaded the new models, and now they align with the architecture proposed by OpenAI for CLIP. I've tested a portion of them in the web interface, and everything is working fine. You can find them here https://huggingface.co/GeorgeBredis

@Natyren is the only thing that you changed when converting the checkpoints to update the activation function? In that case I can just update the config attributes of the `facebook` checkpoints on the hub.

Hello @NielsRogge! I also thought that would be sufficient (as mentioned above), but for the sake of certainty, I decided to verify it. It turned out that the results of the models in such a situation differ (such behavior remained a mystery to me), so I decided to reconvert the weights. However, in theory, the only thing that should differ between the versions is the configuration settings.

Ok I've updated all checkpoints. https://huggingface.co/models?other=metaclip

Hi @NielsRogge , when loading the tokenizers i get error:
```
ValueError: Non-consecutive added token '
' found. Should have index 49408 but has index 49406 in saved vocabulary.
```

Hi,

I'm not able to reproduce that:

```
>>> from transformers import AutoProcessor
>>> processor = AutoProcessor.from_pretrained("facebook/metaclip-b32-400m")
Downloading (…)rocessor_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 504/504 [00:00<00:00, 149kB/s]
Downloading (…)okenizer_config.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 950/950 [00:00<00:00, 494kB/s]
Downloading (…)olve/main/vocab.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.06M/1.06M [00:00<00:00, 8.97MB/s]
Downloading (…)olve/main/merges.txt: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 525k/525k [00:00<00:00, 5.93MB/s]
Downloading (…)in/added_tokens.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 57.0/57.0 [00:00<00:00, 60.4kB/s]
Downloading (…)cial_tokens_map.json: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 133/133 [00:00<00:00, 154kB/s]
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
```

@NielsRogge upgrading the transformers version made it works, Thanks!

Hello @NielsRogge!

This is my very first time deciding to contribute to open source projects inspired by my participation in the Hugging Face event in Paris and the insightful conversations I had with the project maintainers.

As a final-year graduate student in Math and AI, I am eager to explore opportunities to collaborate on this issue. I would greatly appreciate it if you could provide more information on how I can get involved.

Thank you in advance.

Closing this issue since it has been resolved. @mhdirnjbr I'd recommend taking a look at other "good first issues" or "good second issues", you could also contribute a Transformer-based model to the library

@nicolas-dufour What's the version number of transformers you upgraded to? I have tried 4.32.1 and 4.33.3. Both versions gave me the exact same error like yours. Thanks.

Update: Have to quit `conda install` and use `pip install tokenizers transformers` and the issue is gone after the upgrade. Here are the HuggingFace package versions `fsspec-2023.12.2 huggingface_hub-0.20.2 safetensors-0.4.1 tokenizers-0.15.0 transformers-4.36.2`