Llama4 config is not compatible with model initialization
Author: fingertap
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.51.3                                                                   
- Platform: Linux-5.15.0-126-generic-x86_64-with-glibc2.35      
- Python version: 3.12.9                                                                               
- Huggingface_hub version: 0.30.2                                                         
- Safetensors version: 0.5.3                                                                
- Accelerate version: 1.6.0                                                     
- Accelerate config:    not found                                   
- DeepSpeed version: not installed                                                        
- PyTorch version (GPU?): 2.6.0+cu124 (True)                     
- Tensorflow version (GPU?): not installed (NA)                      
- Flax version (CPU?/GPU?/TPU?): not installed (NA)                          
- Jax version: not installed                                                               
- JaxLib version: not installed                                                           
- Using distributed or parallel set-up in script?: Yes                           
- Using GPU in script?: Yes                                          
- GPU type: NVIDIA H200

### Who can help?

When initializing the model with:

```python
config = AutoConfig.from_pretrained("/checkpoint/llama4-scout-16e")
model = Llama4ForCausalLM(config)
```

this error occurs

```
AttributeError: 'Llama4Config' object has no attribute 'hidden_size'
```

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
config = AutoConfig.from_pretrained("/checkpoint/llama4-scout-16e")
model = Llama4ForCausalLM(config)
```

### Expected behavior

No error reported.

Comments:
@fingertap You can fix the loading with the following:
```python
# I assume that's a local path?, checked with the official repo `meta-llama/Llama-4-Scout-17B-16E-Instruct`
config = AutoConfig.from_pretrained("/checkpoint/llama4-scout-16e").get_text_config()
model = Llama4ForCausalLM(config)
```

This is mainly due to llama4 having multimodal capabilities (which makes transformers expect the model to be loaded that way). I'm unsure if this behaviour is wanted or if it should be more lenient. cc @zucchini-nlp 

Thanks!

IMO, when user passes a Llama4Config to Llama4ForCausalLM, he actually means to build the language model without vision model. We can raise a warning about this. However, at least, this should not throw any errors, cuz there is no other explicit way to obtain the text config (I have no experience on MM models, not sure if the `get_text_config` is a must. Even if this is the case, I still think there should be a warning about this and silently call the `get_text_config` for the user).

In fact, initializing with config is common, as we have to init the model on meta device, yet directly calling `from_pretrained` will produce annoying tqdm process bar.

Yes, the ``CausalLM` model expects a text config if initialized without `from_pretrained`. When user load with `from_pretrained` we obtain the necessary config under the hood. I agree that it is not intuitive currently for users and we are planning to clean up auto modeling for multimodal models. Currently, I'd suggest to use `get_text_config()` when init model from config 

I will start working on the multimodal auto mapping next week

Thanks! Based on the information you provided, I recommend at least casting the error into a more intuitive one, with a direct error message telling users that they should use `get_text_config`. Currently the error message (Attribute Error) has nothing to do with the real cause.

Closing this as this is not a bug. Can you guys mention this issue in the following PRs, if any?