LlavaNextForConditionalGeneration - incorrect way of creating `final_labels` in `_merge_input_ids_with_image_features`
Author: heli-qi
Pull request: no
Labels: name: bug

### System Info

# System Info
- `transformers` version: 4.42.4
- Platform: Linux-5.4.0-42-generic-x86_64-with-glibc2.35
- Python version: 3.10.14
- Huggingface_hub version: 0.23.5
- Safetensors version: 0.4.3
- Accelerate version: 0.27.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA GeForce RTX 3090

# Who can help?
@zucchini-nlp @ArthurZucker



### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

# Intention
I'm finetuning LlaVA-Next model by my own dataset. I'm using the PreTrainedModel class `LlavaNextForConditionalGeneration` and the pretrained model `llava-hf/llava-v1.6-vicuna-7b-hf`. I observed a very large loss values at the beginning of training, i.e., larger than 15 for cross-entropy loss.

# Errors
I found that the [`_merge_input_ids_with_image_features`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llava_next/modeling_llava_next.py#L806C17-L814C18) method called in forward() of `LlavaNextForConditionalGeneration` yields an irregular `labels` where the positions of CLIP image features are not properly filled by the `ignore_index` (-100) but the `bos_token_id` (1). I guess the problem may be caused by the behavior how the `final_labels` is created [inside `_merge_input_ids_with_image_features`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llava_next/modeling_llava_next.py#L603) 
```
final_labels = None
if labels is not None:
    labels = labels.to(target_device)
    final_labels = torch.full_like(final_attention_mask, ignore_index).to(torch.long)
    final_labels[batch_indices, text_to_overwrite] = labels[batch_indices, non_image_indices]
```
where `final_attention_mask` is a bool tensor, so `final_labels` becomes a tensor full of 1 after being converted into long type. 
I think the solution could be using `final_input_ids` as the base to create `final_labels` like
```
final_labels = torch.full_like(final_input_ids, ignore_index).to(torch.long)
``` 


### Expected behavior

I hope you kindly help me solve this problem. If there is anything related to LlaVA-Next that I don't know, please let me know and I will do further investigation.

Comments:
@heli-qi hey!

Attention masks are usually in `int64` not booleans. I ran one forward pass with inputs and labels to verify that labels are not cast to boolean `0/1`. 

Can you share a small reproducer of how you're doing fine-tuning?

@zucchini-nlp Many thanks for your rapid response! It turns out that I mistakenly give my attention_mask in torch.bool type in my datacollator. I'm kinda a newbee for HuggingFace APIs and really appreciate your advice.

By the way, just a tiny advice, would it be troublesome that some type checking could be added into forward() for attention_mask like arguments to force their type to torch.long?

We expect you to use tokenizer/processor classes that handle correct typing internally and there is more information about input args in the documentation, so I would say casting every input to its correct type is a bit extra here. 

If you have any further questions, please post them in the [forum](https://discuss.huggingface.co/latest). We try to reserve GH for bugs and feature requests :)

I think we do type hint, but we support booloan and non boolean (4D causal mask for example) on a lot of models, which is why the type hints probably say torch.tensor