SmolVLM2 cannot be used quantized
Author: anderlem
Pull request: no
Labels: name: bug

### System Info

When loading SmolVLM2 quantized with a BitsAndBytesConfig and in bf16 and training, I get 
`RuntimeError: Index put requires the source and destination dtypes match, got BFloat16 for the destination and Float for the source.` at 
```
  File "lib/python3.12/site-packages/transformers/models/smolvlm/modeling_smolvlm.py", line 648, in inputs_merger
image_embeds[image_mask] = image_hidden_states[block_idx[image_mask], local_idx[image_mask], :]
```
image_hidden_states seems not to match the dtype of image_embeds. I fixed it by inserting `image_hidden_states = image_hidden_states.to(dtype=inputs_embeds.dtype)` above that line.

### Who can help?

@yonigozlan @molbap

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
bnb_config: BitsAndBytesConfig = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)
AutoModelForImageTextToText.from_pretrained(
            "HuggingFaceTB/SmolVLM2-2.2B-Instruct",
            quantization_config=bnb_config,
            torch_dtype=torch.bfloat16
)
```

### Expected behavior

No error.

Comments:
Hi @anderlem, I tried reproducing the `bfloat16` RuntimeError but haven't been able to trigger it on my system. This usually means there might be differences in our env or package versions.

To help pinpoint the problem, could you please run the following Python script and share the output?
```
import torch
import transformers
import bitsandbytes
import accelerate

print(f"PyTorch version: {torch.__version__}")
print(f"Transformers version: {transformers.__version__}")
print(f"BitsAndBytes version: {bitsandbytes.__version__}")
print(f"Accelerate version: {accelerate.__version__}")

if torch.cuda.is_available():
    print("CUDA is available: Yes")
    print(f"CUDA version used by PyTorch: {torch.version.cuda}")
    print(f"GPU device: {torch.cuda.get_device_name(0)}")
else:
    print("CUDA is available: No")
```

@anderlem I want to work on this 


@roysammy To be more precise, this happens during training, and happened with different transformers versions.
```
PyTorch version: 2.7.0+cu126
Transformers version: 4.55.4 
BitsAndBytes version: 0.46.0
Accelerate version: 1.9.0
CUDA is available: Yes                                                                                                                                                                                                                                        CUDA version used by PyTorch: 12.6
```

Hey @anderlem and @omsherikar I can't reproduce, can you try and update to latest release(s) and see if it's fixed? Your script does not break for me

@molbap sure I will try !

I WOULD LIKE TO WORK ON THIS


@anderlem like @molbap mentioned, I also tried reproducing again, but did not encounter the issue. Maybe try upgrading to the latest release(s) and try again? Here are the versions which I am using:
```
PyTorch version: 2.8.0+cu126
Transformers version: 4.56.2
BitsAndBytes version: 0.48.1
Accelerate version: 1.10.1
CUDA is available: Yes
CUDA version used by PyTorch: 12.6
```

Did you only load the model or also finetune it on a dataset? I see the bug during training.

Hi @omsherikar I see you pushed some commits to your fix branch that's great, but we can't merge fixes for non-reproduced bugs, happy to fix what's (provably) broken though! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.