[CLIP] allow loading projection layer in vision and text model
Author: patil-suraj
Pull request: yes

The current vision and text models in clip don't return the image or text projected embeddings and the user needs to load the whole `CLIPModel` to be able to get the vision or text embeddings.

This PR adds `CLIPTextModelWithProjection` and `CLIPVisionModelWithProjection` similar to `CLIPTextModel` and `CLIPVisionModel` but with a projection head.  This will allow using only the related modality model instead of loading the full model or having to write wrappers.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18962). All of your documentation changes will be reflected on that endpoint.

Just to understand better - what is a checkpoint that uses such a projection layer? Is that really part of CLIP or rather of the model built on top of CLIP? Also if one uses the `text_embeds` or `image_embeds` output -> what is the purpose of also having the `pooled_output`?

 Wondering if we should instead create a new head here instead of forcing it into the same class? Or is this an architecture that the official CLIP is using often?



> what is a checkpoint that uses such a projection layer? Is that really part of CLIP or rather of the model built on top of CLIP?

The projection layers are already part of the CLIP model. Those are used to convert the final hidden states of vision and text model into clip embedding space.

https://github.com/huggingface/transformers/blob/a26114777ee1c2802e91bd9cb26a3b39974d52ba/src/transformers/models/clip/modeling_clip.py#L880-L881

The reason we added `CLIPTextModel` and `CLIPVisionModel`, is that users could load text and vision models separately as these individual models can be used in downstream task. But the current design is not optimal, as it does not return final clip embeddings. Those final embeddings are very useful for downstream tasks such retrieval, classification. And now these are also being used in text2image or image2image models.

So if a user needs either the text embeds or vision embeds they need to load the whole clip model, or write a custom wrapper module to include the projection layer (which is what we did for safety checker in diffusers https://github.com/huggingface/transformers/blob/main/src/transformers/models/clip/modeling_clip.py#L880)

> Wondering if we should instead create a new head here instead of forcing it into the same class?

Good point! Then maybe we could add `CLIPTextModelWithProjection` and `CLIPVisionModelWithProjection`. 

> Good point! Then maybe we could add CLIPTextModelWithProjection and CLIPVisionModelWithProjection.

I'd prefer that solution too!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18962). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18962). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18962). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18962). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18962). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18962). All of your documentation changes will be reflected on that endpoint.

Reviving the PR, as there are some model in `diffusers` that will need this soon. As discussed above added `CLIPTextModelWithProjection` and `CLIPVisionModelWithProjection`. @patrickvonplaten , @sgugger would be awesome if you could take a look again :) 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18962). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18962). All of your documentation changes will be reflected on that endpoint.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18962). All of your documentation changes will be reflected on that endpoint.