Florence2 stopped working after upgrade to 4.50.0 ("Unrecognized configuration class")
Author: senarvi
Pull request: no
Labels: name: bug

### System Info

The moment the latest transformers version 4.50.0 appeared in PyPI, Florence2 stopped working. I'm new to Huggingface and I'm following the "how to get started" [example](https://huggingface.co/microsoft/Florence-2-large). All I do is

```python
AutoModelForCausalLM.from_pretrained(
    "microsoft/Florence-2-large", torch_dtype=torch.float16, trust_remote_code=True
)
```

This works with 4.49.0, but with 4.50.0 I get

```
Traceback (most recent call last):
  File "/florence2/detect.py", line 15, in <module>
    AutoModelForCausalLM.from_pretrained("microsoft/Florence-2-large", torch_dtype=torch.float16, trust_remote_code=True)
  File "/usr/local/lib/python3.12/dist-packages/transformers/models/auto/auto_factory.py", line 576, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers_modules.microsoft.Florence-2-large.00d2f1570b00c6dea5df998f5635db96840436bc.configuration_florence2.Florence2LanguageConfig'> for this kind of AutoModel: AutoModelForCausalLM.
Model type should be one of AriaTextConfig, BambaConfig, BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BlenderbotConfig, BlenderbotSmallConfig, BloomConfig, CamembertConfig, LlamaConfig, CodeGenConfig, CohereConfig, Cohere2Config, CpmAntConfig, CTRLConfig, Data2VecTextConfig, DbrxConfig, DiffLlamaConfig, ElectraConfig, Emu3Config, ErnieConfig, FalconConfig, FalconMambaConfig, FuyuConfig, GemmaConfig, Gemma2Config, Gemma3Config, Gemma3TextConfig, GitConfig, GlmConfig, GotOcr2Config, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, GraniteConfig, GraniteMoeConfig, GraniteMoeSharedConfig, HeliumConfig, JambaConfig, JetMoeConfig, LlamaConfig, MambaConfig, Mamba2Config, MarianConfig, MBartConfig, MegaConfig, MegatronBertConfig, MistralConfig, MixtralConfig, MllamaConfig, MoshiConfig, MptConfig, MusicgenConfig, MusicgenMelodyConfig, MvpConfig, NemotronConfig, OlmoConfig, Olmo2Config, OlmoeConfig, OpenLlamaConfig, OpenAIGPTConfig, OPTConfig, PegasusConfig, PersimmonConfig, PhiConfig, Phi3Config, PhimoeConfig, PLBartConfig, ProphetNetConfig, QDQBertConfig, Qwen2Config, Qwen2MoeConfig, RecurrentGemmaConfig, ReformerConfig, RemBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, RwkvConfig, Speech2Text2Config, StableLmConfig, Starcoder2Config, TransfoXLConfig, TrOCRConfig, WhisperConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig, ZambaConfig, Zamba2Config.
```

I tried to check what makes the difference. The reason for the error is that in [_BaseAutoModelClass.from_pretrained()](https://github.com/huggingface/transformers/blob/90e2df5d5544e3624ce711a28717204b7779c2d7/src/transformers/models/auto/auto_factory.py#L560), `has_remote_code` should evaluate to `True`, because if I print the config, it looks like this:

```
Florence2Config {
  "_name_or_path": "microsoft/Florence-2-large",
  "architectures": [
    "Florence2ForConditionalGeneration"
  ],
  "auto_map": {
    "AutoConfig": "microsoft/Florence-2-large--configuration_florence2.Florence2Config",
    "AutoModelForCausalLM": "microsoft/Florence-2-large--modeling_florence2.Florence2ForConditionalGeneration"
  },
  ...
```

But after the version upgrade, `has_remote_code` evaluates to `False`, even though the config, looks like this:

```
Florence2Config {
  "architectures": [
    "Florence2ForConditionalGeneration"
  ],
  "auto_map": {
    "AutoConfig": "microsoft/Florence-2-large--configuration_florence2.Florence2Config",
    "AutoModelForCausalLM": "microsoft/Florence-2-large--modeling_florence2.Florence2ForConditionalGeneration"
  },
...
```



### Who can help?

@amyeroberts  @qubvel 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Install transformers==4.50.0
2. Run
 
```python
AutoModelForCausalLM.from_pretrained(
    "microsoft/Florence-2-large", torch_dtype=torch.float16, trust_remote_code=True
)
```

### Expected behavior

This should not crash.

Comments:
@Rocketknight1 do you have an idea of what has been changed in remote code with the recent release?

cc @gante i believe this is related to our changes in CausalLM 🙃  We forgot about custom code users

I can take a detailed look during weekend, just flagging it for now :)

again. I only just fixed this yday on my machine by pip uninstalling transformers from comfyui python_embed folder, and running Kijai version of the Florence2 nodes again which installed a different transformers version, 49 I think. I think its a clash with transformer version that is getting installed. 

but I just updated comfyui to nightly this morning, and its happened again. I get - "Unrecognized configuration class <class 'transformers_modules.Florence-2-large.configuration_florence2.Florence2LanguageConfig'> for this kind of AutoModel: AutoModelForCausalLM."

will test the fix and report back as it might not be working any more.

If you don't want to go through all sorts of hackery while waiting for a fix, you can just back out to transformers 4.49.0 for the moment.

From your python environment:

python -m pip install --upgrade transformers==4.49.0

When the fix comes out you can just upgrade to the latest version running the command again but removing the version number.