Converting TIMM to HF Vision transformers
Author: prabhuteja12
Pull request: no
Labels: name: Good First Issue

### System Info

- `transformers` version: 4.31.0
- Platform: Linux-5.15.0-1040-aws-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.2
- Accelerate version: 0.22.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@amyeroberts 

Thank you for the tool to convert timm weights to HF ViT implementation compatible ones!  I'm, however, facing issues with converting the `augreg` weights. Running 

`python -m transformers.models.vit.convert_vit_timm_to_pytorch --vit_name vit_base_patch16_224.augreg_in21k_ft_in1k --pytorch_dump_folder_path $MY_MODEL_PATH`

results in an error 

``` 
line 156, in convert_vit_checkpoint
    config.patch_size = int(vit_name[-6:-4])
ValueError: invalid literal for int() with base 10: 't_'
```

The issue stems from the code to infer the input image size and patch size configs which are currently hard coded to exist at specific indices (which doesn't always happen).  Can this be fixed? I have right now patched hacked a version locally that works for this specific `vit_name` example, but a more general fix would be great! 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce the behaviour

1. `python -m transformers.models.vit.convert_vit_timm_to_pytorch --vit_name vit_base_patch16_224.augreg_in21k_ft_in1k --pytorch_dump_folder_path local_dump_folder`



### Expected behavior

A converted `pytorch_model.bin` and corresponding `config` in `local_dump_folder`. 

Comments:
Hey! Would you like to open a pr for this? 🤗 

I might take some time before I can work on this, so if that is ok I can take a stab at this.

Sure! in the mean time anyone in the community can feel free to do this, tagging as a good first issue 

Hi,

That's because the script assumes that the patch size can be inferred from the model name:
```
config.patch_size = int(vit_name[-6:-4])
```
however for `vit_base_patch16_224.augreg_in21k_ft_in1k`, the patch size is not at the characters [-6:-4].

Hence it could make sense to make the conversion script more general, by having a more clever way to get the patch size based on the name, for instance using `model_name.find("patch")`

Hi is anyone assigned to this issue I would like to contribute to it


@prabhuteja12 I want to work on this issue , can you give me some more description about the issue.

Yes, a fix has been proposed in #26353  and is under review.  

Given that the proposed fix is not accepted (in the PR). Can I work on this?

Sure! if you take into account the comment from the other PR would be great 🤗 

hi @ArthurZucker since #26908 has been merged do you think we can close this also #26353 while at it

Yes! Closing 🤗