Allow VLMs to have a correct `base_model`
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, being able to simply call `model.base_model` is a useful feat for models which we didn't support in VLMs

There will be no more weird patter like this after this PR:

```python
            if hasattr(model, "visual"):
                vision_module = model.visual
            elif hasattr(model, "model") and hasattr(model.model, "visual"):
                vision_module = model.model.visual


# Now we can
vision_module = model.base_model.visual
```

Also PR fixes loading base model in Llava-like models which was accidentally deleted. We still need a conversion mapping for legacy checkpoints

```python
# throws bunch of missing keys in main and works with this PR
model  = AutoModel.from_pretrained('llava-hf/llava-1.5-7b-hf') 
```

Comments:
run-slow: cohere2_vision, flava, florence2, gemma3, gemma3n, glm4v, lfm2_vl, llama4, llava, qwen2_vl

This comment contains run-slow, running the specified jobs: 

models: ['models/cohere2_vision', 'models/flava', 'models/florence2', 'models/gemma3', 'models/gemma3n', 'models/glm4v', 'models/lfm2_vl', 'models/llama4', 'models/llava', 'models/qwen2_vl']
quantizations: [] ...

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41589). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: cohere2_vision, flava, florence2, gemma3, gemma3n, glm4v, lfm2_vl, llama4, llava, qwen2_vl



This comment contains run-slow, running the specified jobs: 

models: ['models/cohere2_vision', 'models/flava', 'models/florence2', 'models/gemma3', 'models/gemma3n', 'models/glm4v', 'models/lfm2_vl', 'models/llama4', 'models/llava', 'models/qwen2_vl']
quantizations: [] ...


run-slow: cohere2_vision, flava, florence2, gemma3, gemma3n, glm4v, lfm2_vl, llama4, llava, qwen2_vl




run-slow: cohere2_vision, flava, florence2, gemma3, gemma3n, glm4v, lfm2_vl, llama4, llava, qwen2_vl




run-slow: cohere2_vision, flava, florence2, gemma3, gemma3n, glm4v, lfm2_vl, llama4, llava, qwen2_vl




run-slow: cohere2_vision, flava, florence2, gemma3, gemma3n, glm4v, lfm2_vl, llama4, llava, qwen2_vl




run-slow: cohere2_vision, flava, florence2, gemma3, gemma3n, glm4v, lfm2_vl, llama4, llava, qwen2_vl



Maybe merge main first?

run-slow: cohere2_vision, flava, florence2, gemma3, gemma3n, glm4v, lfm2_vl, llama4, llava, qwen2_vl

This comment contains run-slow, running the specified jobs: 

models: ['models/cohere2_vision', 'models/flava', 'models/florence2', 'models/gemma3', 'models/gemma3n', 'models/glm4v', 'models/lfm2_vl', 'models/llama4', 'models/llava', 'models/qwen2_vl']
quantizations: [] ...


run-slow: cohere2_vision, flava, florence2, gemma3, gemma3n, glm4v, lfm2_vl, llama4, llava, qwen2_vl



run-slow: cohere2_vision, flava, florence2, gemma3, gemma3n, glm4v, lfm2_vl, llama4, llava, qwen2_vl



This comment contains run-slow, running the specified jobs: 

models: ['models/cohere2_vision', 'models/flava', 'models/florence2', 'models/gemma3', 'models/gemma3n', 'models/glm4v', 'models/lfm2_vl', 'models/llama4', 'models/llava', 'models/qwen2_vl']
quantizations: [] ...

Looks like the test failing are same as in main `branch`, and the checkpoints are still loaded correctly without mismatch keys

Now it doesn't touch core modeling anymore, because weight loading was refactored. Instead it fixes loading base model in Llava-like models which was accidentally deleted. We still need a conversion mapping for legacy checkpointss

And as per title, make sure that all VLMs have a correct `base_model_prefix`. I made sure that loading with `model  = AutoModel.from_pretrained('llava-hf/llava-1.5-7b-hf')` doesn't throw errors about missing keys

I will request non core-maintainer review after slow tests ✅ 

run-slow: aria, aya_vision, blt, emu3, flava, gemma3, gemma3n, glm4v, glm4v_moe, got_ocr2, internvl, llama4, llava, llava_next, llava_next_video, llava_onevision

This comment contains `run-slow`, running the specified jobs: 

models: ["models/aria", "models/aya_vision", "models/blt", "models/emu3", "models/flava", "models/gemma3", "models/gemma3n", "models/glm4v", "models/glm4v_moe", "models/got_ocr2", "models/internvl", "models/llama4", "models/llava", "models/llava_next", "models/llava_next_video", "models/llava_onevision"]
quantizations: []

Test failures aren't related, cc @molbap whenever you have a chance 😄 

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19361943870)

✅ No failing test specific to this PR 🎉 !


Yep, I think test can be done and will help also with nudging contribs to use standard modeling format. If the model is supposed to deviate from standard, they can easily skip the test

Will add one today if I get time

Tests in main are broken 😢  waiting for a fix so I can merge

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aria, aya_vision, blt, clvp, cohere2_vision, emu3, flava, florence2, gemma3, gemma3n, glm46v, glm4v, glm4v_moe, got_ocr2, internvl, jetmoe