Add support for Florence-2
Author: hlky
Pull request: yes

# What does this PR do?

This PR adds support for Florence-2.

Compared to the existing [remote code](https://huggingface.co/microsoft/Florence-2-base/blob/main/modeling_florence2.py) the main difference is removal of `MySequential`, removal of `einops.rearrange` and `trunc_normal_` and `DropPath` are copied from `timm`.

Fixes #34155

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
https://github.com/huggingface/transformers/blob/b1d3a52f22ea791e0c6084d9827d33d740675b7c/tests/models/florence2/test_modeling_florence2.py#L259-L353

The skipped tests were failing from this line:

https://github.com/huggingface/transformers/blob/b1d3a52f22ea791e0c6084d9827d33d740675b7c/src/transformers/models/florence2/modeling_florence2.py#L2497-L2499

The model_type is included in the config so I'm not sure why.

https://github.com/huggingface/transformers/blob/b1d3a52f22ea791e0c6084d9827d33d740675b7c/tests/models/florence2/test_modeling_florence2.py#L93

The rest + the slow tests are passing.

Thanks! @Rocketknight1 would you be down to do a first review?

Sure! Taking it today.

Thanks for the review, I'll rework this using modular.

```
Traceback (most recent call last):
  File "utils/check_modular_conversion.py", line 73, in <module>
    non_matching_files += compare_files(modular_file_path, args.fix_and_overwrite)
  File "utils/check_modular_conversion.py", line 53, in compare_files
    generated_modeling_content = convert_modular_file(modular_file_path)
  File "/home/user/transformers/utils/modular_model_converter.py", line 1141, in convert_modular_file
    wrapper.visit(cst_transformers)
  File "/home/user/transformers/.venv/lib/python3.8/site-packages/libcst/metadata/wrapper.py", line 204, in visit
    return self.module.visit(visitor)
  File "/home/user/transformers/.venv/lib/python3.8/site-packages/libcst/_nodes/module.py", line 89, in visit
    result = super(Module, self).visit(visitor)
  File "/home/user/transformers/.venv/lib/python3.8/site-packages/libcst/_nodes/base.py", line 236, in visit
    leave_result = visitor.on_leave(self, with_updated_children)
  File "/home/user/transformers/.venv/lib/python3.8/site-packages/libcst/_visitors.py", line 71, in on_leave
    updated_node = leave_func(original_node, updated_node)
  File "/home/user/transformers/utils/modular_model_converter.py", line 1115, in leave_Module
    self._recursively_add_all_new_needed_functions_in_files()
  File "/home/user/transformers/utils/modular_model_converter.py", line 1101, in _recursively_add_all_new_needed_functions_in_files
    dependency, body, self.all_definitions[dependency], parent=parent
KeyError: 'ValueError'
```

Seems to be some issue with modular conversion.

@hlky we think we know what's happening there. There is a fix open for it, give us a little bit!

@hlky modular conversions should be working now - I pulled in changes from another branch! Please let me know if you encounter any other problems.

To use the updated converter, pull the latest changes and then run `python utils/modular_model_converter.py --files_to_parse src/transformers/models/florence2/modular_florence2.py` to generate the `modelling_florence2.py` file.

Hi @hlky, could you provide an update on the status of this PR?  
Since `timm` is compatible with `transformers`, would it be okay for me to continue integrating Florence-2 into `transformers` based on your work?

> would it be okay for me to continue integrating Florence-2

Hi @ducviet00. That would be great, thank you! 🤗

> Hi @hlky, could you provide an update on the status of this PR? Since `timm` is compatible with `transformers`, would it be okay for me to continue integrating Florence-2 into `transformers` based on your work?

Hi, how are you getting on with this?

No problem if there's no progress, just thought I'd check if you could use any help as I'm using Florence-2 for a project and support would be very helpful for me!

@EFord36 
I have a draft version that works well and is compatible with torch.compile.
However, it depends on #35314.
I'll make a PR soon

In the mean time will review the PR to make sure it's ready! 🚀 

Hi @hlky why was this PR closed? I noticed that the `Florence2ForConditionalGeneration` architecture support still hasn't made it into the latest `transformers` version and attempting to run the model with `transformers` results in the error

```
[VllmSidecar]: 2025-05-12T23:27:57.368Z 1 stderr: async with build_async_engine_client(args) as engine_client: {
  source: "vllm-sidecar",
}
[VllmSidecar]: 2025-05-12T23:27:57.368Z 1 stderr: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ {
  source: "vllm-sidecar",
}
[VllmSidecar]: 2025-05-12T23:27:57.368Z 1 stderr: return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 146, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
    return await anext(self.gen)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 259, in build_async_engine_client_from_engine_args
    mq_engine_client = await asyncio.get_running_loop().run_in_executor(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 59, in run {
  source: "vllm-sidecar",
}
[VllmSidecar]: 2025-05-12T23:27:57.369Z 1 stderr: result = self.fn(*self.args, **self.kwargs) {
  source: "vllm-sidecar",
}
[VllmSidecar]: 2025-05-12T23:27:57.369Z 1 stderr: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/client.py", line 101, in __init__ {
  source: "vllm-sidecar",
}
[VllmSidecar]: 2025-05-12T23:27:57.369Z 1 stderr: self.tokenizer = init_tokenizer_from_configs(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/tokenizer_group.py", line 101, in init_tokenizer_from_configs {
  source: "vllm-sidecar",
}
[VllmSidecar]: 2025-05-12T23:27:57.369Z 1 stderr: return TokenizerGroup(
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/tokenizer_group.py", line 23, in __init__
    self.tokenizer = get_tokenizer(self.tokenizer_id, **tokenizer_config) {
  source: "vllm-sidecar",
}
[VllmSidecar]: 2025-05-12T23:27:57.369Z 1 stderr: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/tokenizer.py", line 242, in get_tokenizer {
  source: "vllm-sidecar",
}
[VllmSidecar]: 2025-05-12T23:27:57.369Z 1 stderr: raise e
  File "/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/tokenizer.py", line 221, in get_tokenizer
    tokenizer = AutoTokenizer.from_pretrained(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/tokenization_auto.py", line 973, in from_pretrained {
  source: "vllm-sidecar",
}
[VllmSidecar]: 2025-05-12T23:27:57.369Z 1 stderr: raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers_modules.microsoft.Florence-2-large.00d2f1570b00c6dea5df998f5635db96840436bc.configuration_florence2.Florence2Config'> to build an AutoTokenizer.
Model type should be one of AlbertConfig, AlignConfig, AriaConfig, BarkConfig, BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BlenderbotConfig, BlenderbotSmallConfig, BlipConfig, Blip2Config, BloomConfig, BridgeTowerConfig, BrosConfig, CamembertConfig, CanineConfig, ChameleonConfig, ChineseCLIPConfig, ClapConfig, CLIPConfig, CLIPSegConfig, ClvpConfig, LlamaConfig, CodeGenConfig, CohereConfig, Cohere2Config, ColPaliConfig, ConvBertConfig, CpmAntConfig, CTRLConfig, Data2VecAudioConfig, Data2VecTextConfig, DbrxConfig, DebertaConfig, DebertaV2Config, DiffLlamaConfig, DistilBertConfig, DPRConfig, ElectraConfig, Emu3Config, ErnieConfig, ErnieMConfig, EsmConfig, FalconConfig, FalconMambaConfig, FastSpeech2ConformerConfig, FlaubertConfig, FNetConfig, FSMTConfig, FunnelConfig, GemmaConfig, Gemma2Config, GitConfig, GlmConfig, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, GPTSanJapaneseConfig, GroundingDinoConfig, GroupViTConfig, HeliumConfig, HubertConfig, IBertConfig, IdeficsConfig, Idefics2Config, Idefics3Config, InstructBlipConfig, InstructBlipVideoConfig, JambaConfig, JetMoeConfig, JukeboxConfig, Kosmos2Config, LayoutLMConfig, LayoutLMv2Config, LayoutLMv3Config, LEDConfig, LiltConfig, LlamaConfig, LlavaConfig, LlavaNextConfig, LlavaNextVideoConfig, LlavaOnevisionConfig, LongformerConfig, LongT5Config, LukeConfig, LxmertConfig, M2M100Config, MambaConfig, Mamba2Config, MarianConfig, MBartConfig, MegaConfig, MegatronBertConfig, MgpstrConfig, MistralConfig, MixtralConfig, MllamaConfig, MobileBertConfig, ModernBertConfig, MoonshineConfig, MoshiConfig, MPNetConfig, MptConfig, MraConfig, MT5Config, MusicgenConfig, MusicgenMelodyConfig, MvpConfig, NemotronConfig, NezhaConfig, NllbMoeConfig, NystromformerConfig, OlmoConfig, Olmo2Config, OlmoeConfig, OmDetTurboConfig, OneFormerConfig, OpenAIGPTConfig, OPTConfig, Owlv2Config, OwlViTConfig, PaliGemmaConfig, PegasusConfig, PegasusXConfig, PerceiverConfig, PersimmonConfig, PhiConfig, Phi3Config, PhimoeConfig, Pix2StructConfig, PixtralVisionConfig, PLBartConfig, ProphetNetConfig, QDQBertConfig, Qwen2Config, Qwen2_5_VLConfig, Qwen2AudioConfig, Qwen2MoeConfig, Qwen2VLConfig, RagConfig, RealmConfig, RecurrentGemmaConfig, ReformerConfig, RemBertConfig, RetriBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, RwkvConfig, SeamlessM4TConfig, SeamlessM4Tv2Config, SiglipConfig, Speech2TextConfig, Speech2Text2Config, SpeechT5Config, SplinterConfig, SqueezeBertConfig, StableLmConfig, Starcoder2Config, SwitchTransformersConfig, T5Config, TapasConfig, TransfoXLConfig, TvpConfig, UdopConfig, UMT5Config, VideoLlavaConfig, ViltConfig, VipLlavaConfig, VisualBertConfig, VitsConfig, Wav2Vec2Config, Wav2Vec2BertConfig, Wav2Vec2ConformerConfig, WhisperConfig, XCLIPConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig, YosoConfig, ZambaConfig, Zamba2Config. {
  source: "vllm-sidecar",
}
```