mismatch error of loading CLIPVisionModelWithProjection
Author: garychan22
Pull request: no

### System Info

Hi, I am trying to load the pretrained image encoder from laion/CLIP-ViT-bigG-14-laion2B-39B-b160k using 

```
clip_vision_model = CLIPVisionModelWithProjection.from_pretrained("laion/CLIP-ViT-bigG-14-laion2B-39B-b160k")
```

and the following error appears

```
RuntimeError: Error(s) in loading state_dict for CLIPVisionModelWithProjection:
        size mismatch for visual_projection.weight: copying a param with shape torch.Size([1280, 1664]) from checkpoint, the shape in current model is torch.Size([512, 1664]).
        You may consider adding `ignore_mismatched_sizes=True` in the model `from_pretrained` method.
```

But I can load the pretrained image encoder using

```
clip_model = CLIPModel.from_pretrained("laion/CLIP-ViT-bigG-14-laion2B-39B-b160k")
clip_vision_model_wo_proj = clip_model.vision_model
```


### Who can help?

@muellerzr @sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

As mentioned in the main body of the issue

### Expected behavior

successfully loading the pretrained image encoder without error

Comments:
cc @younesbelkada and @amyeroberts 

Hi @garychan22, thanks for raising this issue! 

This is happening because the value of `projection_dim` being using in the [CLIPVisionModelWithProjection](https://github.com/huggingface/transformers/blob/908f853688c4d523780797f27f83af3c10418e92/src/transformers/models/clip/modeling_clip.py#L1288) is different from the value being used in [CLIPModel](https://github.com/huggingface/transformers/blob/908f853688c4d523780797f27f83af3c10418e92/src/transformers/models/clip/modeling_clip.py#L995). 

For CLIPModel, the value of `projection_dim` is 1280, which is set in the [model config here](https://huggingface.co/laion/CLIP-ViT-bigG-14-laion2B-39B-b160k/blob/8c7a3583335de4dba1b07182dbf81c75137ce67b/config.json#L9).

For  CLIPVisionModelWithProjection, the value of `projection_dim` is 512, because it isn't specified in the model's [vision_config](https://huggingface.co/laion/CLIP-ViT-bigG-14-laion2B-39B-b160k/blob/8c7a3583335de4dba1b07182dbf81c75137ce67b/config.json#L95) and so takes the [default value](https://github.com/huggingface/transformers/blob/977b2f05d5697f33e51111e4834a127a9a76349f/src/transformers/models/clip/configuration_clip.py#L205C2-L205C2). This is used, instead of the value of in the CLIPModel config, because the configuration class for CLIPVisionModelWithProjection is [CLIPVisionConfig](https://github.com/huggingface/transformers/blob/977b2f05d5697f33e51111e4834a127a9a76349f/src/transformers/models/clip/modeling_clip.py#L1280C2-L1280C2). 

To load the checkpoint without errors, you can specify the projection dim directly when using `from_pretrained`: 

```python
clip_vision_model = CLIPVisionModelWithProjection.from_pretrained(
    "laion/CLIP-ViT-bigG-14-laion2B-39B-b160k", projection_dim=1280
)
```




@amyeroberts thanks for your reply, using the suggested way my code works without problems