Fail loading pretrained weights for  Dinov2ForImageClassification model 
Author: ofirshifman
Pull request: no

### System Info

both on:
transformers             4.32.0
transformers             4.34.0.dev0


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I've encountered a bug with the Dinov2ForImageClassification model from Hugging Face Transformers. As per the provided documentation [here](https://huggingface.co/docs/transformers/main/model_doc/dinov2#transformers.Dinov2ForImageClassification), I've followed the code example using the latest Transformers version. However, when running the code, I encounter an error indicating that the model is performing binary classification instead of the expected ImageNet 1000-way classification.

Here's my code:
```
from transformers import AutoImageProcessor, Dinov2ForImageClassification
import torch
from datasets import load_dataset

# Load a sample image dataset (in this case, 'huggingface/cats-image')
dataset = load_dataset('huggingface/cats-image')
image = dataset['test']['image'][0]

# Load the image processor and the Dinov2ForImageClassification model
image_processor = AutoImageProcessor.from_pretrained('facebook/dinov2-base')
model = Dinov2ForImageClassification.from_pretrained('facebook/dinov2-base')

# Prepare the input and obtain logits
inputs = image_processor(image, return_tensors='pt')
with torch.no_grad():
    logits = model(**inputs).logits

# The expected number of labels for ImageNet classification should be 1000
predicted_label = logits.argmax(-1).item()
```

Regardless of whether I specify num_labels=1000 during model initialization to correct the label dimensions, the following error persists:
```
Some weights of Dinov2ForImageClassification were not initialized from the model checkpoint at facebook/dinov2-base and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```

The issue persists, and I'm unable to utilize the pretrained Dinov2ForImageClassification model for ImageNet 1000-way classification as intended.

### Expected behavior

loading without warning, having 1000-way long output vector, that is representing the correct classification labels of ImageNet.

see more here:
https://discuss.huggingface.co/t/dino2-for-classification-has-wrong-number-of-labels/55027

Comments:
cc @NielsRogge and @amyeroberts 

Hi @ofirshifman,

Thanks for your interest in DINOv2. Note that the error is expected, given that [facebook/dinov2-base](https://huggingface.co/facebook/dinov2-base) is only the base Transformer without any head on top, useful for feature extraction. However Meta just released the linear heads on top, hence I'm adding support for them in the PR above (you can load them using `Dinov2ForImageClassification`). 

They are on the hub:
- https://huggingface.co/facebook/dinov2-small-imagenet1k-1-layer
- https://huggingface.co/facebook/dinov2-base-imagenet1k-1-layer
- https://huggingface.co/facebook/dinov2-large-imagenet1k-1-layer
- https://huggingface.co/facebook/dinov2-giant-imagenet1k-1-layer.

Thanks for uploading it to the hub @NielsRogge! That's great!

I was playing a Little with the code and wondered what is the best practice according to the hub's API to get both the CLS output and the base model output, I can edit my own code to get it, but maybe it worth adding ImageClassifierOutputWithPooling or something similar. 
what do you thing?

You can always get the CLS tokens of whatever layers you want by specifying `output_hidden_states=True`:

```
from transformers import Dinov2ForImageClassification
import torch

model = Dinov2ForImageClassification.from_pretrained("facebook/dinov2-small-imagenet1k-1-layer")

pixel_values = torch.randn(1, 3, 224, 224)

with torch.no_grad():
    outputs = model(pixel_values, output_hidden_states=True)

# get the intermediate hidden states, let's use the last one
hidden_states = outputs.hidden_states
last_hidden_state = hidden_states[-1]

# get the CLS token feature, or the patch features
cls_token_feature = last_hidden_state[:,0,:]
patch_features = last_hidden_state[:,1:,:]
```

Just a small note: `pooler_output` is not exactly `cls_token_feature`, since the model also applies LayerNorm on it.
In order to get exactly the `pooler_output` one should add this to your code:
```
        sequence_output = model.base_model.layernorm(last_hidden_state)
        pooled_output = sequence_output[:, 0, :]
```

Thanks again for the help.