[gemma 3] multimodal checkpoints + AutoModelForCausalLM
Author: gante
Pull request: yes
Labels: name: for patch

# What does this PR do?

Fixes #36683 

Enables `AutoModelForCausalLM` with multimodal checkpoints, to load a decoder-only text model.

E.g. the following now works:
```py
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-4b-it")
model = AutoModelForCausalLM.from_pretrained("google/gemma-3-4b-it", device_map="auto")

inputs = tokenizer(["The quick brown"], return_tensors="pt").to(model.device)
gen_out = model.generate(**inputs, do_sample=False)
print(tokenizer.batch_decode(gen_out, skip_special_tokens=True))
```

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36741). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@zucchini-nlp that's curious, this means the issue is possibly setup or commit-dependent 🤔 

On my machine, the test script above fails both on `main` (`c53d53da89c0617f7dd5a69a2a08e6b1232b35fd`) and on gemma3 release commit (`46350f5eae87ac1d168ddfdc57a0b39b64b9a029`), both on CPU and GPU. Same error as reported by users.

What was the commit you've tried it on?

There are two angles to this question:
1. What we communicate to our users;
2. Whether we should enable it or not.

_______________________
Regarding 1:

In the [release notes](https://github.com/huggingface/transformers/releases) we write "For text-only inputs use Gemma3ForCausalLM for generation to avoid loading the vision tower.", so we expect the model to be used as text-only. Both `gemma3` and `gemma3_text` are in the auto map for `AutoModelForCausalLM`. As such, from our side, we indicate to the user that any `gemma3` model is compatible with `AutoModelForCausalLM`, implicitly (can be used as text-only model) and explicitly (it's part of the auto map).

It is not compatible at the moment, so we must choose whether to enable it or update other parts accordingly (including perhaps exceptions) 🤗 

______________________
Regarding 2:

If `gemma3` supports text-only, I'm deeply in favor of enabling it with `AutoModelForCausalLM`:
- `Gemma3ForCausalLM` is supported, and it is the same (minus the ease of use)
- Many downstream libraries use auto classes and don't want to bother with model-specific imports. It makes their development cycle much easier.
- Without `AutoModelForCausalLM`, the alternative auto class loads the vision tower (= waste of memory)
- The changes are minimal, as we can see in this PR

I don't think we should do it for all combinations of modalities, not unless there is a demand for it. However, text-only generation is the most used task in our library and on the hub, and it is expected that new models will often be text + other modalities. As such, if we don't enable this, downstream libraries/users will have to pick between model-specific imports and additional memory usage.

For me, the advantages are very clear :)


Regarding the communication, I see how there might have been some confusion. The idea was that Gemma3 has a 1B checkpoint trained on text-only, so users are expected to load it with `CausalLM`

> If Gemma3 supports text-only, I'm deeply in favor of enabling it with AutoModelForCausalLM  

I understand the reasoning behind this, but I think we should also consider consistency with other VLMs. Gemma3 and Mllama are only models that have a separate `CausalLM` class. Even though any other VLMs can work in text-only setting with decent quality, we didn't add a `CausalLM` class for them since the models released didn't have a text-only checkpoint. If we decide to introduce `CausalLM` for all models that support text, it would make sense to extend the same approach to other VLMs as well.  

As a reference point, LLaVa users have been running the model in text-only mode by loading `ConditionalModel`, and we’ve enabled it by ensuring the processor doesn’t raise errors when `images=None` which is standard by now for all processors afair

> Without AutoModelForCausalLM, the alternative auto class loads the vision tower (= waste of memory)  

Valid concern, though in practice, the additional memory overhead is relatively small (200-500M params with SigLIP). While not ideal, this may not significantly impact users, especially since they are intentionally choosing a multimodal model for text-only tasks

> Many downstream libraries use auto classes and don't want to bother with model-specific imports. It makes their development cycle much easier

I think this is the strongest argument for adding `CausalLM`. However, I’d like to clarify whether this is an actual limitation of VLLM—where it can't generate from text-only inputs with `Gemma3ForConditionalGeneration`—or if this is more about user expectations shaped by our documentation. Given that previous multimodal releases didn’t raise similar concerns, I suspect this is more of a communication issue than technical limitation on VLLM.  

If we decide to add add `CausalLM` for gemma3 multimodal models, we would need to ensure consistency across other model. This shouldn’t be too difficult since part of the standardization process for VLLM already involves separating `BaseModel` and `lm-head` for all multimodals - though it may add some redundancy imho

Would love to hear thoughts on this! If you agree with the above, I would be pro of shaping better our communication next time, and be more explicit on docs. I feel like in general the docs for multimodality lacks a lot of information, but I wanted first to make sure we have a consistent API before adding a section for `Multimodal LLMs`

Thanks a lot for this PR 🙏 . Any chance this can be added to the v4.49.0-Gemma-3 tag? Or is 4.50 around the corner anyway?

>Or is 4.50 around the corner anyway?

yes, it is 😉