Implement DINO V2
Author: Lime-Cakes
Pull request: no
Labels: name: New model

### Model description

Code and model is available here: https://github.com/facebookresearch/dinov2

Full paper here: https://arxiv.org/abs/2304.07193

The implementation seems fairly simple. Most layers is already implemented within transformers library (it's just a ViT). There's some changes compared to DINO (which is implemented already), such as SwiGLU and LayerScale. According to #20403, SwiGLU is already implemented, though, the original code uses xformers's SwiGLU. 

DINO V2 also have a different license as listed here: https://github.com/facebookresearch/dinov2/blob/main/LICENSE
It is NonCommercial.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

_No response_


If there's no issue with license, I can make a PR for the model.

Comments:
Same as issue as mentioned at #23739 

> Same as issue as mentioned at #23739

Oops, I didn't notice. You want to port the weight/code over? Will likely have to add a small layer or two to transformer library and write a weight convert script just like https://github.com/huggingface/transformers/blob/main/src/transformers/models/vit/convert_dino_to_pytorch.py

> > Same as issue as mentioned at #23739
> 
> 
> 
> Oops, I didn't notice. You want to port the weight/code over? Will likely have to add a small layer or two to transformer library and write a weight convert script just like https://github.com/huggingface/transformers/blob/main/src/transformers/models/vit/convert_dino_to_pytorch.py

If you don't mind I'd like to take this one. Also, thanks for the tips I'll take a look at the reference you mentioned 

> > > Same as issue as mentioned at #23739
> > 
> > 
> > Oops, I didn't notice. You want to port the weight/code over? Will likely have to add a small layer or two to transformer library and write a weight convert script just like https://github.com/huggingface/transformers/blob/main/src/transformers/models/vit/convert_dino_to_pytorch.py
> 
> If you don't mind I'd like to take this one. Also, thanks for the tips I'll take a look at the reference you mentioned

Sure, please do. I look forward to it! If there's any new layer implemented, will you also add the corresponding flax implementation for those new layer?

Added [here](https://github.com/huggingface/transformers/pull/24016)! Although on top of it, today Meta released some additional dinov2 models for semantic image segmentation and depth estimation: https://twitter.com/MetaAI/status/1697233915331879329, and it looks like currently, transformers only supports [feature extraction](https://github.com/huggingface/transformers/blob/9c5acca0028b550e1328ba7e2f16418fe0a0c634/src/transformers/models/dinov2/modeling_dinov2.py#L587) and [image classification](https://github.com/huggingface/transformers/blob/9c5acca0028b550e1328ba7e2f16418fe0a0c634/src/transformers/models/dinov2/modeling_dinov2.py#L676).

cc @NielsRogge @fxmarty

(should we close this issue and open a new one?)


Seems like a new issue should be open. This one should be close since DINOv2 PR is merged. I just forgot.