timm to pytorch conversion for vit model fix
Author: staghado
Pull request: yes

This PR fixes this issue #26219 with timm to PyTorch conversion. It removes the need for hard coded values for model dims by using the attributes of the timm model without needing the model name.

It does the following things :

- [x] Extract model dims from the timm model directly, no need for ifs
- [x] Decides whether the converted model will be a classification model or only a feature extractor using the num_classes attribute of the timm model.
- [x] In the case of a feature extractor only model : remove the pooling layers from the PyTorch model and compare the output to the last hidden state instead.

This works for a large number of models in the ViT family.

@ArthurZucker, @amyeroberts, @rwightman 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26908). All of your documentation changes will be reflected on that endpoint.

Overall looking much better, should be more robust now.

A few things, there are a number of vit configurations supported in timm that are not, to my knowledge, supported in transformers. Should there be an attempt to detect? Thinking of some examples
* if `fc_norm` is present (norm after pooling)
* use of global average pooling in combination (or without) class token
* non-overlapping position and class token embedding
* CLIP style vit with `norm_pre` layer present
* SigLIP style vit with `attn_pool` layer present
* and soon, use of 'registers' via `reg_token` param
* use of layer scale in ViT model blocks

Hey! Make sure to rebase to only have your changes! 😉 

> Hey! Make sure to rebase to only have your changes! 😉

I have reset the branch's history and left only my changes which fix the issue here.

> Overall looking much better, should be more robust now.
> 
> A few things, there are a number of vit configurations supported in timm that are not, to my knowledge, supported in transformers. Should there be an attempt to detect? Thinking of some examples
> 
> * if `fc_norm` is present (norm after pooling)
> * use of global average pooling in combination (or without) class token
> * non-overlapping position and class token embedding
> * CLIP style vit with `norm_pre` layer present
> * SigLIP style vit with `attn_pool` layer present
> * and soon, use of 'registers' via `reg_token` param
> * use of layer scale in ViT model blocks

I have tried to add some checks before trying to convert the model from timm to huggingface.
 Checks to be added :
    1. non-overlapping position and class token embedding 
    2. use of 'registers' via `reg_token` param
    3. check when a model has a convolution feature extractor like ResNet50

I have tested the script on the pre-trained ViTs and only the following give errors:

- vit_base_r50_s16_224.orig_in21k (contains a resnet block)
- vit_base_r50_s16_384.orig_in21k_ft_in1k (contains a resnet block)
- vit_small_r26_s32_224.augreg_in21k
- vit_small_r26_s32_224.augreg_in21k_ft_in1k
- vit_small_r26_s32_384.augreg_in21k_ft_in1k
- vit_tiny_r_s16_p8_224.augreg_in21k
- vit_tiny_r_s16_p8_224.augreg_in21k_ft_in1k
- vit_tiny_r_s16_p8_384.augreg_in21k_ft_in1k


@staghado looking good, those hybrid resnet-vit models should be possible to catch (see if below) with a meaningful error .. other than looks ready to go 

`if not isinstance(model.patch_embed, timm.layers.PatchEmbed)` ...

@ArthurZucker