Different weights for XCLIP when loading from configuration
Author: alejopaullier96
Pull request: no

### System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.37.0
- Platform: Linux-5.15.133+-x86_64-with-glibc2.31
- Python version: 3.10.13
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.26.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.2 (True)
- Tensorflow version (GPU?): 2.15.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.5 (gpu)
- Jax version: 0.4.23
- JaxLib version: 0.4.23.dev20240116
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@ni

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am trying to create video embeddings using the `XCLIPVisionModel ` and following the tutorial that can be found [here](https://huggingface.co/docs/transformers/en/model_doc/xclip#transformers.XCLIPVisionModel). First, I loaded the model as shown in the tutorial:
```
processor = AutoProcessor.from_pretrained("microsoft/xclip-base-patch32")
model = XCLIPVisionModel.from_pretrained("microsoft/xclip-base-patch32")
```
The output embeddings from this model are meaningful and correct, as similar-looking videos have high cosine similarity (~1.0) and dissimilar videos have low cosine similarity. 

However, I later loaded the model using a different method via `XCLIPVisionConfig` in the following way (also as shown in the [tutorial](https://huggingface.co/docs/transformers/en/model_doc/xclip#transformers.XCLIPVisionConfig)):
 ```
 from transformers import XCLIPVisionModel, XCLIPVisionConfig

# Initializing a XCLIPVisionModel with microsoft/xclip-base-patch32 style configuration
configuration = XCLIPVisionConfig()

# Initializing a XCLIPVisionModel model from the microsoft/xclip-base-patch32 style configuration
model = XCLIPVisionModel(configuration)

# Accessing the model configuration
configuration = model.config
```

Now the embeddings produced by this model are not meaningful and incorrect (all embeddings have cosine similarity ~1.0 regardless of the video). This is the only piece of code I change in my pipeline.

I consequently think that there might be a problem with weight initialization. 

Any help on the matter is appreciated.

### Expected behavior

Regardless of the method used to load the model it should output the same embeddings.

Comments:
@NielsRogge I intended to tag you but somehow it tagged the wrong person :(

Hi that's how the Transformers library works ;) you can either randomly initialize a model using the configuration, or initialize a model with pre-trained weights using `from_pretrained`. I know this may be confusing at first, but those are the 2 main ways to initialize a model.

We actually did an effort to make this more clear in the docs (by adding the comment "randomly initializing a model" instead of "initializing a model", cc @stevhliu). We could update this for X-CLIP as well.

> We actually did an effort to make this more clear in the docs (by adding the comment "randomly initializing a model" instead of "initializing a model"

Check out the PyTorch block linked [here](https://huggingface.co/docs/transformers/create_a_model#model)! Would you like to open a PR to the XCLIP [docs](https://huggingface.co/docs/transformers/en/model_doc/xclip#x-clip) to clarify the weights are randomly initialized if you're using the configuration?

@NielsRogge @stevhliu thanks both for your quick reply. Now it's clear. Just for clarification, I got a bit confused by this line in the `XCLIPVisionConfig` description:
> Instantiating a configuration with the defaults will yield a similar configuration to that of the X-CLIP [microsoft/xclip-base-patch32](https://huggingface.co/microsoft/xclip-base-patch32) architecture.

I incorrectly assumed that by loading the configuration I would load a model that is _architecture + weights_, while in reality is just only _architecture_.