Add support for open_clip
Author: apolinario
Pull request: no
Labels: name: New model

### Feature request

Add `open_clip` (https://github.com/mlfoundations/open_clip) support to Transformers 

### Motivation

open_clip has released ViT-B-32, ViT-B/16, ViT-B/16+, ViT-L/14 trained on LAION-400M and LAION-2B which are very relevant models - matching and sometimes surpassing OAI models benchmarks - but are not yet compatible with Transformers. 

Also, [soon a ViT-H is going to drop](https://twitter.com/EMostaque/status/1558851591469400066) which will be the SOTA open source CLIP (since OAI never open sourced their ViT-H used to train DALL-E 2) - so it will also make it even more relevant to support OpenCLIP models and code

cc @rwightman

Comments:
(Is the "new model" tag adequate here or it would be considered adapting to the existing CLIP model?) 

cc'ing @patil-suraj. From the README, it seems that they replaced OpenAI's `quickgelu` with `torch.nn.GELU`, which is apparently better.

Normally the strategy is to add a new model, no matter how small the changes to an existing model.

@apolinario @NielsRogge 

@rom1504 and were discussing moving OpenCLIP LAION trained weights to the HF hub under the LAION org https://huggingface.co/laion  ... for enhanced visibility. It'd be nice if the PyTorch model.bin could be shared with the OpenCLIP use (so remap if the HF transformers keys are a bit different). 

I can't comment on the 'add a new model bit', seems unecessary for adding a changeable activation but not a big deal either way. However, there will may be some (small) architecture additions in future LAION + OpenCLIP model releases so requiring yet another model for those would be, well a bit exessive. They will all be done in a manner that can be dynamically enabled/disabled without breaking backwards weight compat.

Romain and I are currently training larger models on LAION-2B (english). I'm using remainder of JUWELS research grant for an L/14, Romain is working via Stability on H/14 and possibly g/14. We've both run into stability problems at this data + model scale, hence the 'might need arch' additions. The checkpoints for H/14 and g/14 will be 3.7G and 5G respectively, so one model hub instance per model would also avoid some waste here :) 

Also, since OpenCLIP is a relatively small and unknown project, it would be nice to keep some pointers and links back there for people looking to train from scratch and/or fine-tune the models.

EDIT re the 'visibility', and LAION org, we've been working on this paper and will likely do a splash once the next revision is out and we get some more results https://openreview.net/forum?id=M3Y74vmsMcY


Hey @rwightman, excited to hear you'd like to contribute Open CLIP to `transformers`!

The implementation of `CLIP` is done using the `ACT2FN` activation function dictionary: https://github.com/huggingface/transformers/blob/a26c752353a127ba8e4728413806f545718a8d78/src/transformers/models/clip/modeling_clip.py#L281

If this is the only change necessary, then it should be loadable directly in the existing architecture by specifying the appopriate `hidden_act` configuration option.

Do you have in mind what other changes might be needed down the road for the support of additional checkpoints? I would be personally be open to having an `OpenCLIP` model archigtecture which could be the host for current checkpoints and upcoming checkpoints, even while unaware of the changes that might need to be done in the future (therefore with modeling code that would be a bit more dynamic than others), but I'm pinging @patrickvonplaten and @sgugger for their opinion as well.

Agreed with @LysandreJik . The change of activation function does not require a new model by itself (since you can set the right one in the config) but if you anticipate other modeling tweaks, a new architecture definitely makes sense.

I was hoping to have transformers CLIP, OpenCLIP, and timm (for vision backbone finetune) all use the same hub weights ... ie something like,
* `https://huggingface.co/laion/vit_base_patch32_laion400m`
* `https://huggingface.co/laion/vit_base_patch32_laion2b`
etc

For `timm`, I'll just remux the weight into timm vit style on the fly. I was thinking the weights would natively match their source (ie OpenCLIP, which is just OpenAI model names w/o any jit / typing mess). Is there any precedent for doing remap of a pytorch bin file on the fly in transformers or are hub weights always native without any on the fly conversion?



No, there is no on-the-fly conversion in Transformers. The state dict is loaded as is.

Closing as support has been added, see e.g. https://huggingface.co/laion/CLIP-ViT-B-32-laion2B-s34B-b79K

> cc'ing @patil-suraj. From the README, it seems that they replaced OpenAI's quickgelu with torch.nn.GELU, which is apparently better.
> 
> Normally the strategy is to add a new model, no matter how small the changes to an existing model.

Should the default `hidden_act` be `gelu` then in `CLIPConfig`?

@fxmarty changing the default activation in the config would be way too breaking though :-)

Hi! We have a similar issue; we want to bring a CLIP model we fine-tuned using Open AI's CLIP [implementation](https://github.com/openai/CLIP/blob/main/clip/model.py) to the Hub. As far as I understand, the two implementations do not match 1-to-1, so... is there any public script to readapt the weights?

I am asking here since it seems related. Let me know if it's better to open a new issue.

P.S. In the process of analysis of the two models, we also noticed that our model, as well as Open AI's weights (e.g., on [azure](https://openaipublic.azureedge.net/clip/models/40d365715913c9da98579312b702a82c18be219cc2a73407c4526f58eba950af/ViT-B-32.pt)), weigh 300MB while HF's checkpoints weigh 600MB. Do you know why?

@g8a9 checkpoint size differences are likely either due to train checkpoint (ie incl optimizer state) vs state dict only, or one is float32 and the other is float16 (since it's exactly 2x I'm guessing the latter).

Original OpenAI -> Transformers conversion code is in transformers https://github.com/huggingface/transformers/blob/main/src/transformers/models/clip/convert_clip_original_pytorch_to_hf.py ... it's error prone though, so be careful, the params are just being copied so if you have any mismatch in sizing it will fail silently.

I have a modified conversion script as a gist that I used to convert OpenCLIP models to Transformers (the ViT OpenCLIP models w/ standard text tower match OpenAI checkpoint naming). It uses copy_ so you get an error if param sizes don't match, but it was hacked together so I manually plugged each model config in.

https://gist.github.com/rwightman/c79fd0241ed3c860e898114931c07990



> Original OpenAI -> Transformers conversion code is in transformers https://github.com/huggingface/transformers/blob/main/src/transformers/models/clip/convert_clip_original_pytorch_to_hf.py ... it's error prone though, so be careful, the params are just being copied so if you have any mismatch in sizing it will fail silently.

Interesting, I usually use `model.load_state_dict` with the default `strict=True` to make sure any missing or unexpected keys as well as size mismatches are caught when porting models. I can add a similar script to convert OpenCLIP models to Transformers if you want.

> I have a modified conversion script as a gist that I used to convert OpenCLIP models to Transformers (the ViT OpenCLIP models w/ standard text tower match OpenAI checkpoint naming). It uses copy_ so you get an error if param sizes don't match, but it was hacked together so I manually plugged each model config in.
> 
> https://gist.github.com/rwightman/c79fd0241ed3c860e898114931c07990

Thanks @rwightman ! We'll start tweaking from here.