Add LLaVA-OneVision-1.5 model and related configurations
Author: g1050
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->
This PR adds **LLaVA-OneVision-1.5** 
Fixes #41081


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @zach-huggingface, @SunMarc and @qgallouedec
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @SunMarc @zach-huggingface
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
cc @zucchini-nlp

Hey @g1050 !  

Nice addition, I suggest to add the model with modular (see [docs](https://huggingface.co/docs/transformers/en/modular_transformers)) to make it easier to review and maintain. I believe llava-onevision 1.5 has a lot of similarities with other LlaVa series models, and modular will help you to reduce LOC in the PR and iterate faster :)

I see the PR went to `ready for review`. Ping me when it is actually ready for review and the CI is green :)

Hi @zucchini-nlp, it’s ready for review when you have a chance. Thanks!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41095). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @zucchini-nlp, thanks for your review~

I’ve addressed your review comments point by point and this should be ready for your review again. The main changes are:
1. Refactored the modeling code to use the `modular` utilities wherever possible, making it easier to reuse and modify.
2. Re-enabled the previously skipped but necessary CI test cases and fixed the failures, including: `test_can_init_all_missing_weights`, `test_feed_forward_chunking`, `test_generate_from_inputs_embeds_1_beam_search`
3. Renamed all Vision model classes from `Rice` to `LlavaOnevision1_5Vision`, and fixed the docs (`LlavaOnevision1_5Config`) instead of skipping.
4. Simplified the `CLS/RoPE handling` without creating extra empty tensor in `LlavaOnevision1_5VisionPretrainedModel.forward`.
5. Various minor fixes.

If there’s anything else that needs changes, please ping me anytime🤗.


Thanks, reviewing today-tomorrow

Hi, @zucchini-nlp @Cyrilvallez. The comments have been addressed, I think... and this should be ready for final review. If there’s anything else that needs changes, please ping me anytime🤗.


Dear @Cyrilvallez  , apologies for the gentle nudge. Whenever convenient, could you please perform the final review? All feedback has been addressed and CI is passing. Thank you very much for your time and guidance.


@Rocketknight1 @ArthurZucker @zucchini-nlp Sorry for the gentle nudge. Whenever convenient, could you please take a look for a final review/approval? All feedback has been addressed and CI is green. If there’s anything else you’d like me to adjust, I’m happy to update right away. Thank you very much for your time and guidance!

run-slow: llava_onevision1_5

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, llava_onevision1_5

This comment contains `run-slow`, running the specified jobs: 

models: ["models/llava_onevision1_5"]
quantizations: []

Hey @anxiangsir! Deeply deeply sorry for the delay on this! We are really focused on the upcoming v5 currently, which explain my late answer on this. Next iterations will be much quicker now that things are calming down.
In the meantime, do you mind fixing the small issues that popped up when I merged the current main branch into the PR? Simply reapplying modular conversion should be enough I believe!

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19641877811)

### Model CI Report

#### ❌ Failed tests

- [llava_onevision1_5](https://github.com/huggingface/transformers/actions/runs/19641877811/job/56247934289):
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_can_load_with_global_device_set
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_attn_implementation_composite_models
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_bc_torch_dtype
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_can_load_with_device_context_manager
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_can_use_safetensors
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_cannot_load_with_meta_device_context_manager
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_cpu_offload
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_disk_offload_safetensors
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_generate
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_00_fp16_pad_left_sdpa_kernels
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_01_fp16_pad_left
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_02_fp16_pad_left_no_attn_mask_sdpa_kernels
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_03_fp16_pad_left_no_attn_mask
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_04_fp16_pad_right_sdpa_kernels
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_05_fp16_pad_right
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_06_fp16_pad_right_no_attn_mask_sdpa_kernels
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_07_fp16_pad_right_no_attn_mask
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_08_fp32_pad_left_sdpa_kernels
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_09_fp32_pad_left
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_10_fp32_pad_left_no_attn_mask_sdpa_kernels
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_11_fp32_pad_left_no_attn_mask
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_12_fp32_pad_right_sdpa_kernels
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_13_fp32_pad_right
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_14_fp32_pad_right_no_attn_mask_sdpa_kernels
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_15_fp32_pad_right_no_attn_mask
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_16_bf16_pad_left_sdpa_kernels
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_17_bf16_pad_left
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_18_bf16_pad_left_no_attn_mask_sdpa_kernels
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_19_bf16_pad_left_no_attn_mask
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_20_bf16_pad_right_sdpa_kernels
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_21_bf16_pad_right
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_22_bf16_pad_right_no_attn_mask_sdpa_kernels
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_23_bf16_pad_right_no_attn_mask
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_eager_matches_sdpa_inference_24_fp32_pad_left_output_attentions
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_load_save_without_tied_weights
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_model_weights_reload_no_missing_tied_weights
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_save_load
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_sdpa_can_compile_dynamic
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_sdpa_can_dispatch_composite_models
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_sdpa_can_dispatch_on_flash
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationModelTest::test_tied_weights_keys
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationIntegrationTest::test_small_model_integration_test
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationIntegrationTest::test_small_model_integration_test_batch
    tests/models/llava_onevision1_5/test_modeling_llava_onevision1_5.py::LlavaOnevision1_5ForConditionalGenerationIntegrationTest::test_small_model_integration_test_batch_different_resolutions