Add FastVLM
Author: camilla-deckard
Pull request: yes
Labels: name: New model

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds FastVLM from Apple. The model's architecture is very similar to LlaVA, the main difference is that it uses a very fast hybrid encoder called FastViTHD. Timm's FastViT implementation is used and the LlaVA modality connector has been slightly modified.

Addresses (https://github.com/huggingface/transformers/issues/38765)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@zucchini-nlp @ariG23498 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @zach-huggingface, @SunMarc and @qgallouedec
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
cc @zucchini-nlp

@kamila-chay is this ready for review?

@jackzhxng sorry for the delay, I had trouble with GPU access, today/tomorrow I should be able to wrap everything up and it will be ready :)

Hi @zucchini-nlp, we can start the review process, everything is ready except for the tests. I'm writing them now but I think I can do that in the meantime while responding to any comments from your end:)

@zucchini-nlp I've addressed all of your comments :)
Also while writing tests, I noticed some places where the existing code could be improved, mainly by checking if submodels support a specific attention  implementation. I saw that we already have this in some tests , e.g.

https://github.com/huggingface/transformers/blob/1d651c749e0a89743025211b9211e87908018c70/tests/test_modeling_common.py#L3590-L3594

but it's skipped in some other cases, so I added it. I also added [those checks](https://github.com/huggingface/transformers/pull/41112/files#diff-a6ebb0bccfe159ca855e9d9b5d6fbbeb62fced0c0296de0993ba39ef5c862cf0R1417-R1420). 

With those 2 modifications more tests pass/are skipped automatically both for FastVLM and also for similar models like PerceptionLM (I didn't change anything on remote but I checked that locally - could also push it here or add another PR if the changes are approved). 



> could also push it here or add another PR if the changes are approved

Yep, would be better since it is general test improvement. We can delete manual overriden skips on existing tests prob, if there were any

run-slow: fast_vlm

This comment contains run-slow, running the specified jobs: 

models: ['models/fast_vlm']
quantizations: [] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41112). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker @Cyrilvallez it's been over a month, pinging you in case you forgot 😊

I did I am sorry

run-slow: auto, fast_vlm

This comment contains `run-slow`, running the specified jobs: 

models: ["models/auto", "models/fast_vlm"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19574673964)

### Model CI Report

#### ❌ Failed tests

- [fast_vlm](https://github.com/huggingface/transformers/actions/runs/19574673964/job/56057510655):
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_can_load_with_global_device_set
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_attn_implementation_composite_models
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_bc_torch_dtype
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_can_load_with_device_context_manager
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_can_use_safetensors
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_cannot_load_with_meta_device_context_manager
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_00_fp16_pad_left_sdpa_kernels
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_01_fp16_pad_left
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_02_fp16_pad_left_no_attn_mask_sdpa_kernels
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_03_fp16_pad_left_no_attn_mask
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_04_fp16_pad_right_sdpa_kernels
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_05_fp16_pad_right
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_06_fp16_pad_right_no_attn_mask_sdpa_kernels
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_07_fp16_pad_right_no_attn_mask
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_08_fp32_pad_left_sdpa_kernels
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_09_fp32_pad_left
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_10_fp32_pad_left_no_attn_mask_sdpa_kernels
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_11_fp32_pad_left_no_attn_mask
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_12_fp32_pad_right_sdpa_kernels
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_13_fp32_pad_right
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_14_fp32_pad_right_no_attn_mask_sdpa_kernels
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_15_fp32_pad_right_no_attn_mask
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_16_bf16_pad_left_sdpa_kernels
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_17_bf16_pad_left
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_18_bf16_pad_left_no_attn_mask_sdpa_kernels
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_19_bf16_pad_left_no_attn_mask
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_20_bf16_pad_right_sdpa_kernels
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_21_bf16_pad_right
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_22_bf16_pad_right_no_attn_mask_sdpa_kernels
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_23_bf16_pad_right_no_attn_mask
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_24_fp32_pad_left_output_attentions
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_from_pretrained_no_checkpoint
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_load_save_without_tied_weights
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_model_base_model_prefix
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_model_weights_reload_no_missing_tied_weights
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_save_load
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_sdpa_can_dispatch_composite_models
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationModelTest::test_tied_weights_keys
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationIntegrationTest::test_generation_no_images
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationIntegrationTest::test_small_model_integration_test
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationIntegrationTest::test_small_model_integration_test_batch



@kamila-chay we can merge when the CI turns green, The weight tying failures aren't related to the model specifically, I will rebase and see if it was fixed on main already. There was a huge refactor recently and the main branch is a bit unstable

upd: We need to run `make fix-copies` to update the model first

@bot /style

Style fix runs successfully without any file modified.

oh wow, style bot doesn't run fix copies, didn't know about that 🥲 

Okk let me look at this issue and fix it, i've been away for a few days and didn't see

A simple fix-copies might fix it all :)

run-slow: fast_vlm

This comment contains `run-slow`, running the specified jobs: 

models: ["models/fast_vlm"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19644165683)

### Model CI Report

#### ❌ Failed tests

- [fast_vlm](https://github.com/huggingface/transformers/actions/runs/19644165683/job/56255827710):
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationIntegrationTest::test_generation_no_images
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationIntegrationTest::test_small_model_integration_test
    tests/models/fast_vlm/test_modeling_fast_vlm.py::FastVlmForConditionalGenerationIntegrationTest::test_small_model_integration_test_batch



**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, fast_vlm

Everything's green now, slow tests should be ok too :) @zucchini-nlp 

run-slow: fast_vlm

This comment contains `run-slow`, running the specified jobs: 

models: ["models/fast_vlm"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19855357053)

✅ No failing test specific to this PR 🎉 !


Thanks, great work @kamila-chay ! Let's merge 🚀