pooler of dino-v2 is newly initialized when loading the pre-trained model
Author: garychan22
Pull request: no

### System Info

I have encountered the following warning when loading `facebook/dinov2-giant`:

`Some weights of Dinov2Model were not initialized from the model checkpoint at models--facebook--dinov2-giant and are newly initialized:['pooler.dense.bias', 'pooler.dense.weight']`

does this mean that we cannot utilize the output.pooler_output as the global feature?

transformers 4.32.0.dev0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

import os 
from transformers import AutoImageProcessor, AutoModel
from PIL import Image

image_path = 'COCO_val2014_000000000285.jpg'
image = Image.open(image_path)

processor = AutoImageProcessor.from_pretrained(r"models--facebook--dinov2-giant")
model = AutoModel.from_pretrained(r"models--facebook--dinov2-giant")

inputs = processor(images=[image] * 2, return_tensors="pt")
outputs = model(**inputs)
last_hidden_states = outputs.last_hidden_state # bs x 257 x 1536, all features [global, patch ....]
global_feature = outputs.pooler_output # bs x 1536, global image feature after pooling layer

print(last_hidden_states.size())

### Expected behavior

expect the outputs.pooler_output is meaningful

Comments:
cc @younesbelkada 

Hi, dear HF staffs. 

I found a similar issue when I am playing with the DINO v1 model. 

[dino-vits16](https://huggingface.co/facebook/dino-vits16)

The following warning message came out after the loading process -

```
Some weights of ViTModel were not initialized from the model checkpoint at /data/models/dino/dino-vits16 and are newly initialized: ['pooler.dense.bias', 'pooler.dense.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

The DINOv2 models work fine since it is fixed in #25392 .

@ValerioL29 Would you like to open a PR to fix this for dino v1?

> @ValerioL29 Would you like to open a PR to fix this for dino v1?

Yes, for sure. I'm glad to help! 🤗

> Hi, dear HF staffs.
> 
> I found a similar issue when I am playing with the DINO v1 model.
> 
> [dino-vits16](https://huggingface.co/facebook/dino-vits16)
> 
> The following warning message came out after the loading process -
> 
> ```
> Some weights of ViTModel were not initialized from the model checkpoint at /data/models/dino/dino-vits16 and are newly initialized: ['pooler.dense.bias', 'pooler.dense.weight']
> You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
> ```
> 
> The DINOv2 models work fine since it is fixed in #25392 .

@NielsRogge Hi Niels, I have done some research on the issue. I found that the transformers' implementation of ViT is kinda different from the original implementation of DINO. I printed out the modules -

* 🤗transformers `ViTModel`

```python
from transformers import ViTModel
model = ViTModel.from_pretrained('facebook/dino-vits16')
print(model)

'''
ViTModel(
  (embeddings): ViTEmbeddings(
    (patch_embeddings): ViTPatchEmbeddings(
      (projection): Conv2d(3, 384, kernel_size=(16, 16), stride=(16, 16))
    )
    (dropout): Dropout(p=0.0, inplace=False)
  )
  (encoder): ViTEncoder(
    (layer): ModuleList(
      (0-11): 12 x ViTLayer(
        (attention): ViTSdpaAttention(
          (attention): ViTSdpaSelfAttention(
            (query): Linear(in_features=384, out_features=384, bias=True)
            (key): Linear(in_features=384, out_features=384, bias=True)
            (value): Linear(in_features=384, out_features=384, bias=True)
            (dropout): Dropout(p=0.0, inplace=False)
          )
          (output): ViTSelfOutput(
            (dense): Linear(in_features=384, out_features=384, bias=True)
            (dropout): Dropout(p=0.0, inplace=False)
          )
        )
        (intermediate): ViTIntermediate(
          (dense): Linear(in_features=384, out_features=1536, bias=True)
          (intermediate_act_fn): GELUActivation()
        )
        (output): ViTOutput(
          (dense): Linear(in_features=1536, out_features=384, bias=True)
          (dropout): Dropout(p=0.0, inplace=False)
        )
        (layernorm_before): LayerNorm((384,), eps=1e-12, elementwise_affine=True)
        (layernorm_after): LayerNorm((384,), eps=1e-12, elementwise_affine=True)
      )
    )
  )
  (layernorm): LayerNorm((384,), eps=1e-12, elementwise_affine=True)
  (pooler): ViTPooler(
    (dense): Linear(in_features=384, out_features=384, bias=True)
    (activation): Tanh()
  )
)
'''
```

* Following the [DINO github repo](https://github.com/facebookresearch/dino?tab=readme-ov-file#pretrained-models-on-pytorch-hub) guides and load it from torch hub

```python
import torch
model = torch.hub.load('facebookresearch/dino:main', 'dino_vits16')
print(model)

'''
VisionTransformer(
  (patch_embed): PatchEmbed(
    (proj): Conv2d(3, 384, kernel_size=(16, 16), stride=(16, 16))
  )
  (pos_drop): Dropout(p=0.0, inplace=False)
  (blocks): ModuleList(
    (0-11): 12 x Block(
      (norm1): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
      (attn): Attention(
        (qkv): Linear(in_features=384, out_features=1152, bias=True)
        (attn_drop): Dropout(p=0.0, inplace=False)
        (proj): Linear(in_features=384, out_features=384, bias=True)
        (proj_drop): Dropout(p=0.0, inplace=False)
      )
      (drop_path): Identity()
      (norm2): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
      (mlp): Mlp(
        (fc1): Linear(in_features=384, out_features=1536, bias=True)
        (act): GELU(approximate='none')
        (fc2): Linear(in_features=1536, out_features=384, bias=True)
        (drop): Dropout(p=0.0, inplace=False)
      )
    )
  )
  (norm): LayerNorm((384,), eps=1e-06, elementwise_affine=True)
  (head): Identity()
)
'''
```

The difference is that the original implementation of the DINO backbone does not contain a pooler (or head?) after the final layer norm module but with an `nn.Identity` head. I guess that's why the `ViTPooler`'s weights are newly initialized and then trigger the warning message.

Luckily the `ViTModel` has a boolean parameter `add_pooling_layer` to control whether to initialize the `ViTPooler` in the model or not. After setting it to be `False` when loading the pretrained DINO model with the `from_pretrained` method, everything works just fine.

```python
from transformers import ViTModel
model = ViTModel.from_pretrained(
    'facebook/dino-vits16',
    add_pooling_layer=False
)
# No more warning messages about pooler weights are newly initialized.
```

So, I think there's no need to open a new PR, right? Perhaps we can add some notes on this within the model card's code example so the user will know it is by design. 🤩



@ValerioL29 A note on the model card seems like a good solution :) 

> @ValerioL29 A note on the model card seems like a good solution :)

Yeah, I think so! I can open a PR on the Hub 😃

Thanks for looking into this and opening a PR! As I'm not part of the facebook org on the hub I'm not able to merge these PRs. Will reach out to someone at Meta who can merge them.