CLIPVisionModel errors on trying to load openai/clip-vit-base-patch16
Author: minimaxir
Pull request: no

`CLIPVisionModel` errors on trying to load [openai/clip-vit-base-patch16](https://huggingface.co/openai/clip-vit-base-patch16), which was added to HF (using `CLIPModel` for loading `patch16` as the documentation example for that repo works without error)

It appears that the model is architected as the `patch32` config, as the "current model" shape correspond to that config.

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/var/folders/m9/s4s3bdq96pn3dk13fbgpw6rm0000gn/T/ipykernel_2831/1425856315.py in <module>
----> 1 model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch16")

/usr/local/lib/python3.9/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1530                 cls._load_state_dict_into_model_low_mem(model, loaded_state_dict_keys, resolved_archive_file)
   1531             else:
-> 1532                 model, missing_keys, unexpected_keys, mismatched_keys, error_msgs = cls._load_state_dict_into_model(
   1533                     model,
   1534                     state_dict,

/usr/local/lib/python3.9/site-packages/transformers/modeling_utils.py in _load_state_dict_into_model(cls, model, state_dict, pretrained_model_name_or_path, ignore_mismatched_sizes, _fast_init)
   1688         if len(error_msgs) > 0:
   1689             error_msg = "\n\t".join(error_msgs)
-> 1690             raise RuntimeError(f"Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}")
   1691 
   1692         if len(unexpected_keys) > 0:

RuntimeError: Error(s) in loading state_dict for CLIPVisionModel:
	size mismatch for vision_model.embeddings.position_ids: copying a param with shape torch.Size([1, 197]) from checkpoint, the shape in current model is torch.Size([1, 50]).
	size mismatch for vision_model.embeddings.patch_embedding.weight: copying a param with shape torch.Size([768, 3, 16, 16]) from checkpoint, the shape in current model is torch.Size([768, 3, 32, 32]).
	size mismatch for vision_model.embeddings.position_embedding.weight: copying a param with shape torch.Size([197, 768]) from checkpoint, the shape in current model is torch.Size([50, 768]).
```

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.17.0
- Platform: macOS-12.3-x86_64-i386-64bit
- Python version: 3.9.7
- PyTorch version (GPU?): 1.11.0 (False)

### Who can help
@patil-suraj

## To reproduce

```python
from transformers import CLIPVisionModel
model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch16")
```


Comments:
Thank you for reporting this! Looking into it.

Found the issue, `CLIPVisionConfig` does not correctly copy the vision arguments from the `CLIPConfig`. It uses the default values., which are defined for the patch32 model.

A quick fix to get this working for now is to load `CLIPConfig`, retrieve the `vision_config` from it and pass it to `from_pretrained`

```python
from transformers import CLIPVisionModel, CLIPConfig
config = CLIPConfig.from_pretrained("openai/clip-vit-base-patch16")
model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch16", config=config.vision_config)
```