Add support for Florence-2
Author: ducviet00
Pull request: yes
Labels: name: New model

# What does this PR do?

Florence-2 is a VLM model, which always in top 10 most downloaded VLMs on huggingface.

This PR continues https://github.com/huggingface/transformers/pull/34160, adds support for Florence-2.

Fixes:
- https://github.com/huggingface/transformers/issues/34155
- https://github.com/kijai/ComfyUI-Florence2/issues/151

# Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.



Comments:
@Rocketknight1  @ArthurZucker  PTAL 🤗  


Sorry for the delay! @cyrilvallez can you take the review? Ping me if you need me to take any of it

I refactored the Vision module to use SDPA and FlashAttention, but the outputs differ from those of the eager attention implementation — it seems to be sensitive 🤔  

```
run_example("<DETAILED_CAPTION>")
# fp32 cpu: The image shows a stop sign sitting on the side of a street in Chinatown, New York City. There is a car driving on the road, bollards, a statue, buildings with windows, name boards, trees, and a clear blue sky in the background.
# fp32 gpu: The image shows a stop sign sitting on the side of a street in Chinatown, New York City. There is a car driving on the road, bollards, a statue, buildings with windows, name boards, trees, and a clear blue sky in the background.
# fp16 gpu: The image shows a stop sign sitting on the side of a street in Chinatown, New York City. There is a car driving on the road, bollards, a statue, buildings with windows, name boards, trees, and a clear blue sky in the background.
# sdpa image attention + sdpa language attention
# fp16 gpu: The image shows a stop sign sitting on the side of a street in front of a building. There is a car driving on the road, and in the background there are buildings, trees, and a clear blue sky.
# eager image attention + sdpa language attention
# fp16 gpu: The image shows a stop sign sitting on the side of a street in Chinatown, New York City. There is a car driving on the road, bollards, a statue, buildings with windows, name boards, trees, and a clear blue sky in the background.
```



I need to refactor `processing` files and ~~utilize `timm` for vision backbone~~

When using `load_backbone` for vision models, I realized it breaks the [single model, single file](https://huggingface.co/blog/transformers-design-philosophy) philosophy.
So, should it be copied from timm instead?

Need your comments @Rocketknight1  @ArthurZucker  @Cyrilvallez 

@zucchini-nlp  Thanks for your comments ❤️  I will fix them soon 

cc @zucchini-nlp should I review this? 

@ArthurZucker 
sorry for the delay, I've been busy, so the PR isn't complete yet. I still need to:
- overwrite `_prepare_encoder_decoder_kwargs_for_generation` - waiting for review
- refactor `Processor` class - waiting for review
- write unittests  for `Processor`
- create a script to convert weights from the original format to hf format - waiting for review

👋  Hey! Just pushed a new version with the `_prepare_encoder_decoder_kwargs_for_generation` override for generation. Would love a review of the modeling code when you get a chance @zucchini-nlp  @ArthurZucker 🤗 
Planning to make the conversion script today!

Reviewing today, or the latest on Monday :)

@zucchini-nlp  @ArthurZucker 
Just FYI, I've converted the original model weights to HuggingFace format. The outputs are identical to the original checkpoint when using eager attention, however sdpa attention yields different outputs.

I believe this is because the vision model uses sdpa instead of eager attention (unlike the original), and the model is sensitive to small changes in vision features. Since vision features pass through the language encoder then decoder, there's a butterfly effect where minor differences get amplified.

However, the output quality remains as good as the original - just not identical. The model still passes the `test_eager_matches_sdpa_inference` tests

Removing my review as I will be waiting for @zucchini-nlp to ping me for that! 🤗 

damm I just missed v4.54.0 😭 
I've pushed tests for the processor. Seems all comments have been resolved. Could you review them again when you have time @zucchini-nlp ?

Reviewing now. Don't worry, for new models we will add a preview-tag such as `v4.54-Florence-preview`

run-slow: florence2

This comment contains run-slow, running the specified jobs: 

models: ['models/florence2']
quantizations: [] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_38188). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: florence2

This comment contains run-slow, running the specified jobs: 

models: ['models/florence2']
quantizations: [] ...

@zucchini-nlp z
In order to pass the slow tests, it needs the new converted models. I'll push new model to a hf hub to run slow tests instead of `microsoft/Florence-2-base`

I pushed converted base model to `ducviet00/Florence-2-base-hf` so integration tests should pass from now on

Should I write integration test for large model? @zucchini-nlp 

All tests passed 

> Should I write integration test for large model?

If not yet written, yes. A few tests for generation is enough



Added integration tests for large model

Use this script to convert model: 
```
python convert_florence2_original_pytorch_to_hf.py \
		--hf_model_id microsoft/Florence-2-large \
		--pytorch_dump_folder_path florence2-large \
		--output_hub_path ducviet00/Florence-2-large-hf
```

run-slow: auto, florence2

This comment contains run-slow, running the specified jobs: 

models: ['models/auto', 'models/florence2']
quantizations: [] ...


@zucchini-nlp Just FYI — I formatted Florence-2's docs following [[this issue](https://github.com/huggingface/transformers/issues/36979)](https://github.com/huggingface/transformers/issues/36979).
Also made `ImageTextToTextPipeline` work out of the box with a small patch: set `skip_special_tokens=False` by default

> work out of the box with a small patch: set skip_special_tokens=False by default

I wonder why is this? 

@zucchini-nlp
because for tasks that detect locations, setting `skip_special_tokens=True` removes all `<locXX>` tokens, which are needed for post-processing.

Ah you mean in `post_process_image_text_to_text`, oke!

To merge this PR, is there anything I need to update on my side? 🤗 ❤️ 
@zucchini-nlp  @Cyrilvallez 

No, we are waiting for core maintainer's review now (from Cyril) 😄 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

Morning @Cyrilvallez, I’ve resolved your comments. Could you review it again when you have time? 🤗 



**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

Just split out `get_placeholder_mask` as helper: https://github.com/huggingface/transformers/pull/39777

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

@zucchini-nlp 
I see BART doesn't support `_supports_attention_backend`. Could you please briefly explain why, and how it could be enabled? I'd love to contribute! :)


@ducviet00 Bart already has the new attention API so the only thing we need is to make sure that `kwargs` are propagated from the generative model up to the attention layer. These `kwargs` will hold any optional kwargs used by custom attention types, without which we can't support them

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

Thank you! @zucchini-nlp. I'm still concerned about BART's attention implementation. When I run BART with GPU, it fails the `test_flash_attn_2_inference_equivalence` test, which also causes Florence2 to fail the same test.

cc @vasqu

Seems like there is a bug in fa2 in general, not only BART so taking a look atm - thanks for pinging tho :D

#40002 should fix the fa2 issues 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

Hi @Cyrilvallez 
Please take a look 🤗 

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

`run-slow` locally and passes all tests

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

cc @Cyrilvallez can we merge the model?

run-slow: florence2

This comment contains run-slow, running the specified jobs: 

models: ['models/florence2']
quantizations: [] ...

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

`run-slow` locally and passes all tests but CI failed due to:

> Too long with no output (exceeded 10m0s): context deadline exceeded

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

Luck Is All We Need

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

Thank you for taking a look, @zucchini-nlp
cc @Cyrilvallez in case you missed notification 🤗 


Hi @Cyrilvallez 
Could you take a look at this PR? I'd really appreciate it 🙇 Hopefully, we can finish it this week 🤗 

run-slow: florence2

This comment contains run-slow, running the specified jobs: 

models: ['models/florence2']
quantizations: [] ...

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, florence2

Thank you all for carefully reviewing this PR and for building such a wonderful Transformers library 🤗 
@Cyrilvallez @zucchini-nlp @vasqu 

Hey @ducviet00 , thanks for doing this!

However, I'm currently using the 'large-ft' model https://huggingface.co/microsoft/Florence-2-large-ft , any chance you could upload a version of these weights that's compatible with this MR, or share any code for how you did the conversion?

Really appreciate your work on this!

@EFord36
Hi, you can find it here: 
https://huggingface.co/florence-community

Thanks so much!