Add FastViT model
Author: JorgeAV-ai
Pull request: yes

# What does this PR do?

Fixes #25526 
I have seen that the PR is still open and still no PR has been submitted during these weeks so I have decided to open mine once I finish the model structure + testing 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR. 
cc: @amyeroberts 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerz and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->



Comments:
cc @rafaelpadilla 😉 to keep your eyes on this! 

Small ping here, sorry if this is a bad time for pinging you, i am going to collect some of the doubts that have arisen during this days applying the model structure:

1. This model is characterized by having a different training-test infrastructure, are any other model applied inside of huggingface with this 'feature'? I would like to have a reference but i did not find any model with this 'special feature' (As a quick explanation, basically they train with all the batchnorms + residual connections, once is trained, for inference they restructure the weights, deleting norms + residual connections and some convolutions)
2. The question about the ImageProcessor has not been solved yet, i am going to formulate it again. I am using ViTImageProcessor to avoid repeat the same code, the unique difference is the image size, by default in ViTImageProcessor is 224 and for FastViT is 256, Should I add a "FastViTImageProcessor"? The idea is to keep the code as clear as possible and at the same time as readable as possible, in my opinion is not worth it but i will let you guys decide 👍 .

Thank you so much for your time 
Ping @ArthurZucker @rafaelpadilla
And I am going to ping @NielsRogge , I am pretty sure he can clarify some of the doubts 😊 

Hey! 
1. I am not really sure, but I think the most important is to add support for inference rather than training, especially if they have a very complicated training pipeline, let's just go with inference. 
2. No, you should be able to change the image size when pushing a ` ViTImageProcessor` to the hub! No need for a new model if something can be controlled from a configuration/init argument. 

Hi @JorgeAV-ai regarding creating an image processor: it's pretty important to make sure that the image processor in HF Transformers creates the exact same pixel values as the ones created in the original implementation. Just using ViTImageProcessor will probably result in slightly different pixel_values. Things to be taken into account are the image mean/std when normalizing, whether padding/resizing/center cropping is applied, etc. 

See [here](https://github.com/huggingface/transformers/blob/408b2b3c5057b275855ae4c43c452a7f0b37aa45/src/transformers/models/nougat/convert_nougat_to_hf.py#L179-L183) for an example how we check that both match.

Thanks for the help, the PR is now ready for review. I noticed the reviewer for this PR is @rafaelpadilla , so any feedback would be greatly appreciated 😁

I added some questions above. I also noticed that some of your comments might be related to an outdated code. Would you mind taking a look again? Thanks 😊

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi @JorgeAV-ai are you planning to work further on this PR?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hi，Are you still working on it?

@RobotiX101 As @JorgeAV-ai [didn't reply to this Q](https://github.com/huggingface/transformers/pull/26172#issuecomment-1807829712), I think it's safe to assume the PR is inactive. If you or anyone else would like to try and tackle adding the model we'd be happy to review a PR! 

> @RobotiX101 As @JorgeAV-ai [didn't reply to this Q](https://github.com/huggingface/transformers/pull/26172#issuecomment-1807829712), I think it's safe to assume the PR is inactive. If you or anyone else would like to try and tackle adding the model we'd be happy to review a PR!

OK, I will try.

@amyeroberts Hi, can I take this on, if it's up for grabs to work on and the maintainers are interested in this model to be added?

@RUFFY-369 The work is open for anyone who wishes to to pick up :) We prioritise reviews based on when PRs are open, rather than claims on issues, as we find this helps avoid things going stale 

@amyeroberts Thank you for your reply. I will take a look at this soon after getting completely done with ProPainter :+1: :smile: