`LlavaNextForConditionalGeneration` forward pass broken
Author: qgallouedec
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 5.0.0.dev0 (70179949f7, but it also fails with 4.57.2)
- Platform: Linux-5.15.0-1048-aws-x86_64-with-glibc2.31
- Python version: 3.12.12
- Huggingface_hub version: 1.2.3
- Safetensors version: 0.7.0
- Accelerate version: 1.13.0.dev0
- Accelerate config:    not found
- DeepSpeed version: 0.18.2
- PyTorch version (accelerator?): 2.9.0+cu128 (CUDA)
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA H100 80GB HBM3

### Who can help?

@yonigozlan @molbap @zucchini-nlp 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import torch
from transformers import AutoModelForImageTextToText

model = AutoModelForImageTextToText.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf", dtype="auto")
input_ids = torch.randint(0, model.config.text_config.vocab_size, (2, 8))
model(input_ids=input_ids)
```

```
Traceback (most recent call last):
  File "/fsx/qgallouedec/trl/report.py", line 8, in <module>
    model(input_ids=input_ids)
  File "/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/utils/generic.py", line 764, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/transformers/src/transformers/models/llava_next/modeling_llava_next.py", line 668, in forward
    logits = self.lm_head(hidden_states[:, slice_indices, :])
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/fsx/qgallouedec/miniconda3/envs/trl/lib/python3.12/site-packages/torch/nn/modules/linear.py", line 134, in forward
    return F.linear(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: expected m1 and m2 to have the same dtype, but got: c10::BFloat16 != c10::Half
```

### Expected behavior

Not to fail

Comments:
This issue actually comes from `llava-hf/llava-v1.6-mistral-7b-hf`, see https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf/discussions/46. The following works:

```python
import torch
from transformers import AutoModelForImageTextToText

model = AutoModelForImageTextToText.from_pretrained("llava-hf/llava-v1.6-mistral-7b-hf", dtype="auto", revision="refs/pr/46")
input_ids = torch.randint(0, model.config.text_config.vocab_size, (2, 8))
model(input_ids=input_ids)
```

Duplicate of https://github.com/huggingface/transformers/issues/42641