KeyError: 'llava_qwen2'
Author: fengyuentau
Pull request: no
Labels: name: bug

### System Info

transformers: 4.53.0
os: wsl2
python: 3.12.4

### Who can help?

Hello @amyeroberts @qubvel, I was trying to set up FastVLM inference with SGLang and it prompted me that llava_qwen2 is not supported by transformers. Log is attached below:

```
Traceback (most recent call last):
  File "/root/.pyenv/versions/sglang/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1218, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
                   ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/sglang/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 914, in __getitem__
    raise KeyError(key)
KeyError: 'llava_qwen2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/root/.pyenv/versions/sglang/lib/python3.12/site-packages/sglang/launch_server.py", line 11, in <module>
    server_args = prepare_server_args(sys.argv[1:])
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/sglang/lib/python3.12/site-packages/sglang/srt/server_args.py", line 1748, in prepare_server_args
    server_args = ServerArgs.from_cli_args(raw_args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/sglang/lib/python3.12/site-packages/sglang/srt/server_args.py", line 1691, in from_cli_args
    return cls(**{attr: getattr(args, attr) for attr in attrs})
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 171, in __init__
  File "/root/.pyenv/versions/sglang/lib/python3.12/site-packages/sglang/srt/server_args.py", line 329, in __post_init__
    model_config = ModelConfig.from_server_args(self)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/sglang/lib/python3.12/site-packages/sglang/srt/configs/model_config.py", line 278, in from_server_args
    return ModelConfig(
           ^^^^^^^^^^^^
  File "/root/.pyenv/versions/sglang/lib/python3.12/site-packages/sglang/srt/configs/model_config.py", line 79, in __init__
    self.hf_config = get_config(
                     ^^^^^^^^^^^
  File "/root/.pyenv/versions/sglang/lib/python3.12/site-packages/sglang/srt/utils.py", line 2775, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/sglang/lib/python3.12/site-packages/sglang/srt/hf_transformers_utils.py", line 121, in get_config
    config = AutoConfig.from_pretrained(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/sglang/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1220, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `llava_qwen2` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

You can update Transformers with the command `pip install --upgrade transformers`. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command `pip install git+https://github.com/huggingface/transformers.git`
```

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Install sglang https://docs.sglang.ai/start/install.html
2. Download models from https://github.com/apple/ml-fastvlm?tab=readme-ov-file#model-zoo
3. Run with the following command:

```
python3 -m sglang.launch_server \
  --model-path /path/to/ml-fastvlm/checkpoints/llava-fastvithd_7b_stage2 \
  --host 0.0.0.0 \
  --port 30000
```

### Expected behavior

It should just work. I can make contributions if providing guidance.

Comments:
I don't see that model in the list of [supported models](https://docs.sglang.ai/supported_models/multimodal_language_models.html).

I wouldn't flag this as a bug, rather a feature request if the already existing implementations of llava that you can find on the Transformers library differ from the architecture you're trying to load. Still, I'm unsure if that would also require a change in sglang to make the serving work.

Hello @DWarez , thank you for your information. I want to target on loading model first, then moving serving. Could you point me to a precise documentation on supporting this llava variant?

@fengyuentau can you check the llava-onevision model, it supports checkpoints like "[llava-hf/llava-onevision-qwen2-7b-ov-hf](https://huggingface.co/llava-hf/llava-onevision-qwen2-7b-ov-hf)". 

We have an issue requesting to add the model in https://github.com/huggingface/transformers/issues/38765 and currently we are waiting for more details from Apple team

If you want to use the model with transformers, you need to set `trust_remote_code=True`. See https://github.com/apple/ml-fastvlm/issues/1#issuecomment-2883616294 for a minimal example

Closing the issue as duplicate, feel free to comment under the linked feature request issue

@zucchini-nlp Thank you for the information.

> we are waiting for more details from Apple team

I wonder if there is anything we can do as part of the community instead of just waiting.

cc @ariG23498 if there's anything we can do for the community

I do not have any update from the team. I don't see why people cannot just upload the model weights to their individual repos and use the models. (Just need to be more vigilant about the license)