Bug: Qwen2 vl mrope support?
Author: effortprogrammer
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.47.0.dev0
- Platform: Linux-5.4.0-137-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.26.2
- Safetensors version: 0.4.5
- Accelerate version: 0.34.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: distributed
- GPU type: CUDA GPU

### Who can help?

@ArthurZucker @zucchini-nlp  @amyeroberts

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am trying to further train Qwen/Qwen2-VL-7B-Instruct model and got this error: 

```markdown
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/work/.vscode-server/extensions/ms-python.debugpy-2024.13.2024111901-linux-x64/bundled/libs/debugpy/_vendored/pydevd/pydevd.py", line 3704, in <module>
[rank0]:     main()
[rank0]:   File "/home/work/.vscode-server/extensions/ms-python.debugpy-2024.13.2024111901-linux-x64/bundled/libs/debugpy/_vendored/pydevd/pydevd.py", line 3689, in main
[rank0]:     globals = debugger.run(setup["file"], None, None, is_module)
[rank0]:   File "/home/work/.vscode-server/extensions/ms-python.debugpy-2024.13.2024111901-linux-x64/bundled/libs/debugpy/_vendored/pydevd/pydevd.py", line 2687, in run
[rank0]:     return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)
[rank0]:   File "/home/work/.vscode-server/extensions/ms-python.debugpy-2024.13.2024111901-linux-x64/bundled/libs/debugpy/_vendored/pydevd/pydevd.py", line 2695, in _exec
[rank0]:     globals = pydevd_runpy.run_path(file, globals, "__main__")
[rank0]:   File "/home/work/.vscode-server/extensions/ms-python.debugpy-2024.13.2024111901-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 310, in run_path
[rank0]:     return _run_module_code(code, init_globals, run_name, pkg_name=pkg_name, script_name=fname)
[rank0]:   File "/home/work/.vscode-server/extensions/ms-python.debugpy-2024.13.2024111901-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 127, in _run_module_code
[rank0]:     _run_code(code, mod_globals, init_globals, mod_name, mod_spec, pkg_name, script_name)
[rank0]:   File "/home/work/.vscode-server/extensions/ms-python.debugpy-2024.13.2024111901-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 118, in _run_code
[rank0]:     exec(code, run_globals)
[rank0]:   File "/data/workspace/howard/workspace/Llava/qwen2_vl_instruct_post_train.py", line 423, in <module>
[rank0]:     main(train_args)
[rank0]:   File "/data/workspace/howard/workspace/Llava/qwen2_vl_instruct_post_train.py", line 329, in main
[rank0]:     model = Qwen2VLForConditionalGeneration.from_pretrained(model_name_or_path, config=config)
[rank0]:   File "/data/workspace/howard/workspace/.venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4096, in from_pretrained
[rank0]:     model = cls(config, *model_args, **model_kwargs)
[rank0]:   File "/data/workspace/howard/workspace/.venv/lib/python3.10/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1422, in __init__
[rank0]:     self.model = Qwen2VLModel(config)
[rank0]:   File "/data/workspace/howard/workspace/.venv/lib/python3.10/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1070, in __init__
[rank0]:     self.rotary_emb = Qwen2VLRotaryEmbedding(config=config)
[rank0]:   File "/data/workspace/howard/workspace/.venv/lib/python3.10/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 146, in __init__
[rank0]:     self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
[rank0]: KeyError: 'mrope'
```

While debugging, I realized that inside of the Qwen2VLRotaryEmbedding class, there's ROPE_INIT_FUNCTIONS that checks the rope setting, and process by mapping each function.

However, Qwen2 VL model is using Multimodal Rotary Position Embedding(M-RoPE), which seems not handled currently. Is there any plans,WIP to support this rope setting or am I missing something? 

Current implementation of ROPE_INIT_FUNCTIONS: 

```python
ROPE_INIT_FUNCTIONS = {
    "default": _compute_default_rope_parameters,
    "linear": _compute_linear_scaling_rope_parameters,
    "dynamic": _compute_dynamic_ntk_parameters,
    "yarn": _compute_yarn_parameters,
    "longrope": _compute_longrope_parameters,
    "llama3": _compute_llama3_parameters,
}
```

### Expected behavior

should support mrope.

Comments:
@effortprogrammer hey! mrope is to be honest same as the default implementation and the logic for adding extra dimension (as we have images) is already part of the Qwen2-VL code. TBH the configuration file should have replaced the key `mrope` with `default` when loading, can you share how you are calling the model?


@zucchini-nlp I am calling the model by using this code: 

```python
model = Qwen2VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-VL-7B-Instruct", torch_dtype="auto", device_map="auto"
)

```

When loading model with this, it gave me this error: 

```markdown
[rank0]: Traceback (most recent call last):
[rank0]:   File "/home/work/.vscode-server/extensions/ms-python.debugpy-2024.13.2024111901-linux-x64/bundled/libs/debugpy/_vendored/pydevd/pydevd.py", line 3704, in <module>
[rank0]:     main()
[rank0]:   File "/home/work/.vscode-server/extensions/ms-python.debugpy-2024.13.2024111901-linux-x64/bundled/libs/debugpy/_vendored/pydevd/pydevd.py", line 3689, in main
[rank0]:     globals = debugger.run(setup["file"], None, None, is_module)
[rank0]:   File "/home/work/.vscode-server/extensions/ms-python.debugpy-2024.13.2024111901-linux-x64/bundled/libs/debugpy/_vendored/pydevd/pydevd.py", line 2687, in run
[rank0]:     return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)
[rank0]:   File "/home/work/.vscode-server/extensions/ms-python.debugpy-2024.13.2024111901-linux-x64/bundled/libs/debugpy/_vendored/pydevd/pydevd.py", line 2695, in _exec
[rank0]:     globals = pydevd_runpy.run_path(file, globals, "__main__")
[rank0]:   File "/home/work/.vscode-server/extensions/ms-python.debugpy-2024.13.2024111901-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 310, in run_path
[rank0]:     return _run_module_code(code, init_globals, run_name, pkg_name=pkg_name, script_name=fname)
[rank0]:   File "/home/work/.vscode-server/extensions/ms-python.debugpy-2024.13.2024111901-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 127, in _run_module_code
[rank0]:     _run_code(code, mod_globals, init_globals, mod_name, mod_spec, pkg_name, script_name)
[rank0]:   File "/home/work/.vscode-server/extensions/ms-python.debugpy-2024.13.2024111901-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 118, in _run_code
[rank0]:     exec(code, run_globals)
[rank0]:   File "/data/workspace/howard/workspace/Llava/qwen2_vl_instruct_post_train.py", line 423, in <module>
[rank0]:     main(train_args)
[rank0]:   File "/data/workspace/howard/workspace/Llava/qwen2_vl_instruct_post_train.py", line 329, in main
[rank0]:     model = Qwen2VLForConditionalGeneration.from_pretrained(model_name_or_path, config=config)
[rank0]:   File "/data/workspace/howard/workspace/.venv/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4096, in from_pretrained
[rank0]:     model = cls(config, *model_args, **model_kwargs)
[rank0]:   File "/data/workspace/howard/workspace/.venv/lib/python3.10/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1422, in __init__
[rank0]:     self.model = Qwen2VLModel(config)
[rank0]:   File "/data/workspace/howard/workspace/.venv/lib/python3.10/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 1070, in __init__
[rank0]:     self.rotary_emb = Qwen2VLRotaryEmbedding(config=config)
[rank0]:   File "/data/workspace/howard/workspace/.venv/lib/python3.10/site-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 146, in __init__
[rank0]:     self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
[rank0]: KeyError: 'mrope'
```

Lemme check, that's weird as the official checkpoints were working a while ago

@effortprogrammer which version of transformers you are using? The model loaded without errors for me in the latest version from `main`

@zucchini-nlp I downloaded transformers from github source two days ago, so I believe it's pretty latest version. Should I uninstall current transformers version and re-install from source?

@effortprogrammer no, that is recent enough and it is weird. Can you share what you get after `transformers-cli env` and also run the following code
```python
from transformers import AutoConfig

config = AutoConfig.from_pretrained("Qwen/Qwen2-VL-7B-Instruct")
print(config)
```

The config should have a key `rope_scaling` which is what we need to see

@zucchini-nlp Hey, well my bad.. it was my code issue. it should call the config by AutoConfig, but calling config with different model,, sorry for confusion. Closing this issue for now