Repository: huggingface/transformers
Pull request: Deprecate #36741 and map Causal to Conditional

# What does this PR do?

Fixes #36886, fixes #36926 and loading in SmolAgents (from feedback in internal slack)

NOTE: this is a temp fix. In long term we will converge under one Auto for all multimodals, which we don't have yet. As such we will keep `CausalLM` as temporal dump for custom code users. 🔴 We will break this pattern in the near future and might enforce text-only models under this mapping

After #36741, we unintentionally broke model loading for most remote code users, because many Vision/Audio/Omni LLMs on the hub use `CausalLM` mapping with `AutoTokenizer`. This happens because `AutoImageTextToText` is less visible, and also because we have no mapping for other modalities.

This PR deprecates the previous fix and properly maps Gemma3 4B+ models to its `ConditionalGeneration` class which aligns with info in model card. As discussed internally, all vision-audio-multimodal models will converge under `AutoModelForCausalLM` in the future to maintain consistency and stop adding new mapping for each new modality.

- Why? All Audio LMs are already in causal mapping due to the lack of `AutoAudioTextToText` or `AutoAudioToText` mappings. Vision LMs have also been inconsistently mapped under CausalLM

- Consequences: No breaking changes for users, including those using remote code. A warning is raised only if a model has both full config and text config mapped in `CausalLM` which is a super rare case. (gemma-3 was exception)

- Edge cases: Other models like `llava-1.5` cannot load anymore under `AutoModelForCausalLM` after this PR, but checkpoint keys won’t match anyway + we never got user issues. This was an existing issue, not introduced by this PR. It might be resolved by a bigger refactor for vLLM after adding base models for all VLMs and correct `base-prefix-keys`

I verified that Gemma3 case works as expected, without raising warnings (I just changed mapping class). The only difference from the previous fix is that vision tower is loaded as well, which might affect advanced users who manipulated configs or access model layers manually.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained('googel/gemma-3-4b-it')
tokenizer = AutoTokenizer.from_pretrained('googel/gemma-3-4b-it')
inputs = tokenizer('What is your name?', return_tensors="pt")
out = model.generate(**inputs)
tokenizer.batch_decode(out)
>>> ['<bos>What is your name?\n\nI am called Pixel.\n\nHow are you?\n\nI am functioning optimally, thank you for']

# Chat template
msg = [{"role": "user", "content": "What is your name?"}]
tokenizer.apply_chat_template([msg], tokenize=True, return_dict=True, return_tensors="pt", add_generation_prompt=True)
out = model.generate(**inputs)
tokenizer.batch_decode(out)
>>> ['<bos><start_of_turn>user\nWhat is your name?<end_of_turn>\n<start_of_turn>model\nMy name is Gemma. I was created by the Gemma team at Google DeepMind. \n\nI']
```


Changed files:
filename: docs/source/en/model_doc/gemma3.md
patch: @@ -204,7 +204,7 @@ visualizer("<img>What is shown in this image?")
     +   do_pan_and_scan=True,
         ).to("cuda")
     ```
-- For text-only inputs, use [`AutoModelForCausalLM`] instead to skip loading the vision components and save resources.
+- For Gemma-3 1B checkpoint trained in text-only mode, use [`AutoModelForCausalLM`] instead.
 
     ```py
     import torch

filename: src/transformers/configuration_utils.py
patch: @@ -1118,7 +1118,9 @@ def get_text_config(self, decoder=False) -> "PretrainedConfig":
         Returns the config that is meant to be used with text IO. On most models, it is the original config instance
         itself. On specific composite models, it is under a set of valid names.
 
-        If `decoder` is set to `True`, then only search for decoder config names.
+        Args:
+            decoder (`Optional[bool]`, *optional*, defaults to `False`):
+                If set to `True`, then only search for decoder config names.
         """
         decoder_possible_text_config_names = ("decoder", "generator", "text_config")
         encoder_possible_text_config_names = ("text_encoder",)
@@ -1140,8 +1142,10 @@ def get_text_config(self, decoder=False) -> "PretrainedConfig":
                 "case, using `get_text_config()` would be ambiguous. Please specify the desied text config directly."
             )
         elif len(valid_text_config_names) == 1:
-            return getattr(self, valid_text_config_names[0])
-        return self
+            config_to_return = getattr(self, valid_text_config_names[0])
+        else:
+            config_to_return = self
+        return config_to_return
 
 
 def get_configuration_file(configuration_files: list[str]) -> str:

filename: src/transformers/models/auto/modeling_auto.py
patch: @@ -522,7 +522,7 @@
         ("fuyu", "FuyuForCausalLM"),
         ("gemma", "GemmaForCausalLM"),
         ("gemma2", "Gemma2ForCausalLM"),
-        ("gemma3", "Gemma3ForCausalLM"),
+        ("gemma3", "Gemma3ForConditionalGeneration"),
         ("gemma3_text", "Gemma3ForCausalLM"),
         ("git", "GitForCausalLM"),
         ("glm", "GlmForCausalLM"),
@@ -1671,7 +1671,20 @@ def _prepare_config_for_auto_class(cls, config: PretrainedConfig) -> PretrainedC
         Under the hood, multimodal models mapped by AutoModelForCausalLM assume the text decoder receives its own
         config, rather than the config for the whole model. This is used e.g. to load the text-only part of a VLM.
         """
-        return config.get_text_config(decoder=True)
+        possible_text_config_names = ("decoder", "generator", "text_config")
+        text_config_names = []
+        for text_config_name in possible_text_config_names:
+            if hasattr(config, text_config_name):
+                text_config_names += [text_config_name]
+
+        text_config = config.get_text_config(decoder=True)
+        if text_config_names and type(text_config) in cls._model_mapping.keys():
+            warnings.warn(
+                "Loading a multimodal model with `AutoModelForCausalLM` is deprecated and will be removed in v5. "
+                "`AutoModelForCausalLM` will be used to load only the text-to-text generation module.",
+                FutureWarning,
+            )
+        return config
 
 
 AutoModelForCausalLM = auto_class_update(AutoModelForCausalLM, head_doc="causal language modeling")

filename: tests/models/gemma3/test_modeling_gemma3.py
patch: @@ -344,15 +344,15 @@ def test_flex_attention_with_grads(self):
 
     def test_automodelforcausallm(self):
         """
-        Regression test for #36741 -- make sure `AutoModelForCausalLM` works with a Gemma3 config, i.e. that
+        Regression test for #36741/#36917 -- make sure `AutoModelForCausalLM` works with a Gemma3 config, i.e. that
         `AutoModelForCausalLM.from_pretrained` pulls the text config before loading the model
         """
         config = self.model_tester.get_config()
         model = Gemma3ForConditionalGeneration(config)
         with tempfile.TemporaryDirectory() as tmp_dir:
             model.save_pretrained(tmp_dir)
             for_causal_lm = AutoModelForCausalLM.from_pretrained(tmp_dir)
-            self.assertIsInstance(for_causal_lm, Gemma3ForCausalLM)
+            self.assertIsInstance(for_causal_lm, Gemma3ForConditionalGeneration)
 
 
 @slow

filename: tests/models/mistral3/test_processor_mistral3.py
patch: @@ -20,7 +20,7 @@
 import requests
 
 from transformers import PixtralProcessor
-from transformers.testing_utils import require_vision
+from transformers.testing_utils import require_read_token, require_vision
 from transformers.utils import is_torch_available, is_vision_available
 
 from ...test_processing_common import ProcessorTesterMixin
@@ -35,6 +35,7 @@
 
 
 @require_vision
+@require_read_token
 class Mistral3ProcessorTest(ProcessorTesterMixin, unittest.TestCase):
     """This tests Pixtral processor with the new `spatial_merge_size` argument in Mistral3."""