RuntimeError: Index put requires the source and destination dtypes match, got Half for the destination and Float for the source.
Author: sinngam-khaidem
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.53.0.dev0
- Platform: Linux-5.4.0-113-generic-x86_64-with-glibc2.39
- Python version: 3.10.18
- Huggingface_hub version: 0.33.0
- Safetensors version: 0.5.3
- Accelerate version: 1.7.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.7.0+cu126 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA A100-SXM4-80GB


### Who can help?

@amyeroberts @qubvel @zach-huggingface @SunMarc @zucchini-nlp 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

I was trying to finetune Idefics2 (8B) on custom dataset(that conists of text-image pairs) using the notebook  <a href = "https://colab.research.google.com/drive/1NtcTgRbSBKN7pYD3Vdx1j9m8pt3fhFDB?usp=sharing">Idefics2 Finetuning Tutorial</a> when I ran into the following error. 

<img width="1679" alt="Image" src="https://github.com/user-attachments/assets/1bd79661-0c37-42b8-ab3f-6ddea4ef9277" />

I made small modifications to the hyperparameters for my use case.

### Training Arguments
```python
training_args = TrainingArguments(
    eval_strategy="steps",
    eval_steps=3,
    max_steps = 30,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    gradient_accumulation_steps=4,
    warmup_steps=5,
    learning_rate=2e-4,
    weight_decay=0.01,
    output_dir="output",
    bf16=True,
    remove_unused_columns=False,
    run_name="run01",
    report_to="wandb",
)
```
### Data Collator
```python
class MyDataCollator:
    def __init__(self, processor):
        self.processor = processor
        self.processor.tokenizer.add_special_tokens({
            "additional_special_tokens": ["<end_of_utterance>", "<image>"]
        })
        self.image_token_id = processor.tokenizer.additional_special_tokens_ids[
            processor.tokenizer.additional_special_tokens.index("<image>")
        ]

    def __call__(self, examples):
        texts = []
        images = []
        for example in examples:
            image = example["image"]
            question = example["question"]
            answer = example["answer"]
            messages = [
                {
                    "role": "user",
                    "content": [
                        {"type": "text", "text": question},
                        {"type": "image"},
                    ]
                },
                {
                    "role": "assistant",
                    "content": [
                        {"type": "text", "text": answer}
                    ]
                }
            ]
            text = processor.apply_chat_template(messages, add_generation_prompt=False)
            texts.append(text.strip())
            images.append([image])

        batch = processor(text=texts, images=images, return_tensors="pt", padding=True)

        labels = batch["input_ids"].clone()
        labels[labels == processor.tokenizer.pad_token_id] = self.image_token_id
        batch["labels"] = labels
        return batch
```
How do I fix this?

Comments:
# Solution: Fix Data Type Mismatch in Idefics2 Fine-tuning
The error occurs because there's a mismatch between Half (float16) and Float (float32) data types. Here are several approaches to fix this:
## Option 1: Force Consistent Data Types in Data Collator (Recommended)
Modify your MyDataCollator to ensure all tensors are in the same precision:
```python
class MyDataCollator:
    def __init__(self, processor, use_bf16=True):
        self.processor = processor
        self.use_bf16 = use_bf16
        self.processor.tokenizer.add_special_tokens({
            "additional_special_tokens": ["<end_of_utterance>", "<image>"]
        })
        self.image_token_id = processor.tokenizer.additional_special_tokens_ids[
            processor.tokenizer.additional_special_tokens.index("<image>")
        ]

    def __call__(self, examples):
        texts = []
        images = []
        for example in examples:
            image = example["image"]
            question = example["question"]
            answer = example["answer"]
            messages = [
                {
                    "role": "user",
                    "content": [
                        {"type": "text", "text": question},
                        {"type": "image"},
                    ]
                },
                {
                    "role": "assistant",
                    "content": [
                        {"type": "text", "text": answer}
                    ]
                }
            ]
            text = self.processor.apply_chat_template(messages, add_generation_prompt=False)
            texts.append(text.strip())
            images.append([image])

        batch = self.processor(text=texts, images=images, return_tensors="pt", padding=True)

        # Convert all tensors to the same dtype
        if self.use_bf16:
            batch = {k: v.to(torch.bfloat16) if v.dtype == torch.float32 else v for k, v in batch.items()}
        else:
            batch = {k: v.to(torch.float32) if v.dtype == torch.bfloat16 else v for k, v in batch.items()}

        labels = batch["input_ids"].clone()
        labels[labels == self.processor.tokenizer.pad_token_id] = self.image_token_id
        batch["labels"] = labels
        return batch
```

## Option 2: Disable BF16 Training
If the above doesn't work, try training without BF16:
```python
training_args = TrainingArguments(
    eval_strategy="steps",
    eval_steps=3,
    max_steps=30,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    gradient_accumulation_steps=4,
    warmup_steps=5,
    learning_rate=2e-4,
    weight_decay=0.01,
    output_dir="output",
    bf16=False,  # Disable BF16
    fp16=False,  # Also disable FP16 to be safe
    remove_unused_columns=False,
    run_name="run01",
    report_to="wandb",
)
```

Thanks for reporing! We should fix it and cast `image_features` to the same `dtype` as LM embeddings in the model. The above suggested solution can be used a workaround for now, but it won't fix if LM and ViT have different `dtypes`

I will open a PR :) 

It was fixed by https://github.com/huggingface/transformers/pull/38467, closing as resolved. Note that you need to install from source, as the fix was not included in the latest release