Clip embeddings text/vision missmatch for the model 'laion/CLIP-ViT-H-14-laion2B-s32B-b79K'
Author: johngull
Pull request: no

### System Info

- `transformers` version: 4.26.1
- Platform: Linux-5.15.0-60-generic-x86_64-with-glibc2.35
- Python version: 3.10.9
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I use text and vision models for the same clip model and got different dimensionality of embeddings, which is wrong with the idea of the CLIP models.

Here is the source code to reproduce (outputs as comments)

```
CACHE_PATH = "../models_cache"
from transformers import CLIPFeatureExtractor, CLIPTextModel, CLIPTokenizer, CLIPVisionModel, AutoProcessor, CLIPConfig
clip_model = CLIPTextModel.from_pretrained("laion/CLIP-ViT-H-14-laion2B-s32B-b79K", cache_dir=CACHE_PATH)
clip_model = clip_model.to("cuda")
tokenizer = CLIPTokenizer.from_pretrained("laion/CLIP-ViT-H-14-laion2B-s32B-b79K", cache_dir=CACHE_PATH)

clip_v_model = CLIPVisionModel.from_pretrained("laion/CLIP-ViT-H-14-laion2B-s32B-b79K", cache_dir=CACHE_PATH)
clip_v_model = clip_v_model.to("cuda")
v_preprocess = AutoProcessor.from_pretrained("laion/CLIP-ViT-H-14-laion2B-s32B-b79K", cache_dir=CACHE_PATH)

....
image = Image.open(img_filename)
prompt = "some prompt"

with torch.no_grad():
    inputs = tokenizer([prompt], padding=True, return_tensors="pt").to(gpu_device)
    outputs = clip_model(**inputs)
    print(outputs.pooler_output.shape) # torch.Size([1, 1024])
    print(outputs.last_hidden_state.shape) # torch.Size([1, 12, 1024])

    inputs = v_preprocess(images=image, return_tensors="pt").to(gpu_device)
    image_features = clip_v_model(**inputs)
    print(image_features.pooler_output.shape) # torch.Size([1, 1280]) !!!!
    print(image_features.last_hidden_state.shape) # torch.Size([1, 257, 1280])
```

### Expected behavior

I expect CLIPVisionModel to produce embedding with the shape [1, 1024].

Comments:
cc @amyeroberts 

Hi, this particular model (laion/CLIP-ViT-H-14-laion2B-s32B-b79K) uses a different `hidden_size` for the text and vision encoders (1024 and 1280 respectively), but they get projected to the same dimensionality using a linear projection layer (for this model, the `projection_dim` is 1024 as seen in the [config](https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K/blob/main/config.json#L177)). It's recommended to use the [get_text_features](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPModel.get_text_features) and [get_image_features](https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPModel.get_image_features) methods of `CLIPModel` to get embeddings which have the same dimensionality. The pooler output is pre-projection.

@NielsRogge  Thank you for the explanation