multi_modality error
Author: ashimdahal
Pull request: no
Labels: name: bug

### System Info

traceback (most recent call last):
  File "/home/ashim/miniconda3/envs/captioning/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1092, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
  File "/home/ashim/miniconda3/envs/captioning/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 794, in __getitem__
    raise KeyError(key)
KeyError: 'multi_modality'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ashim/projects/caption_images/main.py", line 116, in <module>
    "decoder": AutoModelForCausalLM.from_pretrained(
  File "/home/ashim/miniconda3/envs/captioning/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 526, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
  File "/home/ashim/miniconda3/envs/captioning/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1094, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `multi_modality` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

You can update Transformers with the command `pip install --upgrade transformers`. If this does not work, and the checkpoint is very new, then there may not be a release version that supports this model yet. In this case, you can get the most up-to-date code by installing Transformers from source with the command `pip install git+https://github.com/huggingface/transformers.git`
(captioning) ashim@usm-research-vm:~/projects/caption_images$ python
Python 3.10.16 (main, Dec 11 2024, 16:24:50) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import transformers
>>> transformers.__version__
'4.50.0.dev0'
>>>

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

    # Qwen-VL - Use original model
    {
        "processor": "Ertugrul/Qwen2-VL-7B-Captioner-Relaxed",
        "decoder": Qwen2ForCausalLM.from_pretrained(
            "Ertugrul/Qwen2-VL-7B-Captioner-Relaxed",
            trust_remote_code=True
        )
    },
    
    # DeepSeek Janus - Use as causal LM
    {
        "processor": "deepseek-ai/Janus-Pro-7B",
        "decoder": AutoModelForCausalLM.from_pretrained(
            "deepseek-ai/Janus-Pro-7B",
            trust_remote_code=True
        )

### Expected behavior

just load the two models as part of the dictionary and move on

Comments:
@ashimdahal did you install the Janus repo first (https://github.com/deepseek-ai/Janus/tree/main)? I recommend to install the repo and follow instructions there. And if it doesn't work, feel free to open an issue in their repo

The model is not yet supported natively by transformers, and we are planning to add it soon

@zucchini-nlp , thanks I noticed that adding just even an unused module like they did on their readme

from janus.models import MultiModalityCausalLM

would clear the problem. Thanks for your feedback. Have a nice day 🤗

Oh I see, importing unused module helped because when importing the register calls were executed, which are located in the same file at the end. Closing as resolved then

cc @Rocketknight1 since it's auto-class related, maybe Janus team isn't using register correctly 🤷🏻