loading CLIPVisionModel from openai/clip-vit-base-patch32
Author: PaulLerner
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.12.5
- Platform: Linux-4.18.0-305.57.1.el8_4.x86_64-x86_64-with-redhat-8.4-Ootpa
- Python version: 3.7.11
- PyTorch version (GPU?): 1.10.0+cu102 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO


### Who can help?

@patil-suraj

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

CLIPVisionModel keys do not match the state dict available in `"openai/clip-vit-base-patch32"`. Perhaps it can be fixed but you should at least change the doc which provides the sample code below: https://huggingface.co/docs/transformers/model_doc/clip#transformers.CLIPVisionModel
```py
>>> from transformers import CLIPVisionModel
>>> model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32")
You are using a model of type clip to instantiate a model of type clip_vision_model. This is not supported for all configurations of models and can yield errors.                                          
Some weights of the model checkpoint at openai/clip-vit-base-patch32 were not used when initializing CLIPVisionModel: ['text_model.encoder.layers.1.layer_norm1.weight', …, 'text_model.encoder.layers.9.self_attn.q_proj.bias']
- This IS expected if you are initializing CLIPVisionModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model fro
m a BertForPreTraining model).
- This IS NOT expected if you are initializing CLIPVisionModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSeq
uenceClassification model).


```

# Circumventing it
I was able to circumvent this problem easily:
```py
>>> from transformers import CLIPVisionModel, CLIPModel
>>> cv = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32")
# ignore warning and load the full CLIP
>>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
# load one state dict into the other and save it
>>> cv.vision_model.load_state_dict(model.vision_model.state_dict())
>>> cv.save_pretrained('/path/of/your/choice')
```

### Expected behavior

```py
>>> from transformers import CLIPVisionModel
>>> model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32")
# all clear
```



Comments:
I don't see the issue here, it is just telling you that the weights of the text encoder aren't used. Which makes total sense as you are loading only the vision encoder, whose weights are all loaded properly.

It's a warning, not an error, so I'm not sure whether this can be improved.

Oh sorry, my bad, the message warning was so huge I thought it also discarded the weights of the vision model.