AttributeError: 'HybridCache' object has no attribute 'float' — PaliGemma2 Evaluation Fails with BF16
Author: iremeyiokur
Pull request: no
Labels: name: bug

name: Cache

### System Info

- `transformers` version: 4.50.0
- Platform: Linux-4.15.0-20-generic-x86_64-with-glibc2.27
- Python version: 3.10.16
- Huggingface_hub version: 0.28.1
- Safetensors version: 0.5.2
- Accelerate version: 1.3.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (GPU?): 2.5.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: Yes
- Using GPU in script?: Yes
- GPU type: NVIDIA RTX A6000

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Steps to reproduce the behavior:

1. Install the latest version of transformers (4.50.0).
2. Load dataset using:
```
from datasets import load_dataset
dataset = load_dataset("imagefolder", data_dir="/path/to/data", split="train")
```
3. Fine-tune Paligemma2 with Trainer using:
```
from transformers import PaliGemmaProcessor, PaliGemmaForConditionalGeneration, TrainingArguments, Trainer, BitsAndBytesConfig

model_id ="google/paligemma2-3b-pt-448"

args = TrainingArguments(
    num_train_epochs=15,
    remove_unused_columns=False,
    per_device_train_batch_size=1,
    per_device_eval_batch_size=1,
    bf16=True,
    output_dir="./output",
    eval_strategy="epoch",
)

trainer = Trainer(
        model=model,
        train_dataset=dataset ,
        eval_dataset=dataset_test,
        data_collator=collate_fn,
        args=args
        )
```

4. Run trainer.evaluate() and observe the error:
`AttributeError: 'HybridCache' object has no attribute 'float'`

Error Traceback:
```
[rank3]: Traceback (most recent call last):
[rank3]:   File "Paligemma2/finetuning.py", line 190, in <module>
[rank3]:     trainer.evaluate()
[rank3]:   File "/miniconda3/envs/vlm/lib/python3.10/site-packages/transformers/trainer.py", line 4136, in evaluate
[rank3]:     output = eval_loop(
[rank3]:   File "miniconda3/envs/vlm/lib/python3.10/site-packages/transformers/trainer.py", line 4330, in evaluation_loop
[rank3]:     losses, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
[rank3]:   File "/miniconda3/envs/vlm/lib/python3.10/site-packages/transformers/trainer.py", line 4546, in prediction_step
[rank3]:     loss, outputs = self.compute_loss(model, inputs, return_outputs=True)
[rank3]:   File "/miniconda3/envs/vlm/lib/python3.10/site-packages/transformers/trainer.py", line 3783, in compute_loss
[rank3]:     outputs = model(**inputs)
[rank3]:   File "/miniconda3/envs/vlm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
[rank3]:     return self._call_impl(*args, **kwargs)
[rank3]:   File "/miniconda3/envs/vlm/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
[rank3]:     return forward_call(*args, **kwargs)
[rank3]:   File "/miniconda3/envs/vlm/lib/python3.10/site-packages/accelerate/utils/operations.py", line 819, in forward
[rank3]:     return model_forward(*args, **kwargs)
[rank3]:   File "/miniconda3/envs/vlm/lib/python3.10/site-packages/accelerate/utils/operations.py", line 807, in __call__
[rank3]:     return convert_to_fp32(self.model_forward(*args, **kwargs))
[rank3]:   File "/miniconda3/envs/vlm/lib/python3.10/site-packages/accelerate/utils/operations.py", line 786, in convert_to_fp32
[rank3]:     return recursively_apply(_convert_to_fp32, tensor, test_type=_is_fp16_bf16_tensor)
[rank3]:   File "/miniconda3/envs/vlm/lib/python3.10/site-packages/accelerate/utils/operations.py", line 118, in recursively_apply
[rank3]:     {
[rank3]:   File "/miniconda3/envs/vlm/lib/python3.10/site-packages/accelerate/utils/operations.py", line 119, in <dictcomp>
[rank3]:     k: recursively_apply(
[rank3]:   File "/miniconda3/envs/vlm/lib/python3.10/site-packages/accelerate/utils/operations.py", line 126, in recursively_apply
[rank3]:     return func(data, *args, **kwargs)
[rank3]:   File "/miniconda3/envs/vlm/lib/python3.10/site-packages/accelerate/utils/operations.py", line 778, in _convert_to_fp32
[rank3]:     return tensor.float()
[rank3]: AttributeError: 'HybridCache' object has no attribute 'float'
```


### Expected behavior

Evaluation should proceed without errors.

Thanks in advance for your help! Let me know if I can provide more details.

Comments:
cc @gante , seems to be caused by recent changes to make Cache inherit from `Tensor` and not `Module` 

+1 also happens with Gemma 3

Hi folks 👋 

I'm aware of the problem, but I haven't worked on a solution. Meanwhile, if you set `use_cache: false` in the model config, you should be able to train the model :)

> Hi folks 👋
> 
> I'm aware of the problem, but I haven't worked on a solution. Meanwhile, if you set `use_cache: false` in the model config, you should be able to train the model :)

Hi, I've tried this using the code below, but I'm still getting the same error :(

`trainer.model.config.use_cache = False`
`trainer.evaluate()`

`trainer.train()` is running without error only if I train the model without evaluation. 

Have to second [iremeyiokur](https://github.com/iremeyiokur), that didn't fix the error for me etiher.

we might also need to set `trainer.model.generation_config.use_cache = False`

The root issue is that `HybridCache` has a `dtype` attribute, which makes `accelerate` treat it as a tensor in `convert_to_fp32` (and thus the attempt to cast it with `float()`) 😭 

Opening a PR to fix it :)

With these changes, I'm able to train with evaluation. Thanks! 

thanks @gante !

excause me, which version of transformers should i use?

@sy-sunmoon you need to install from `main` branch with `!pip install --upgrade git+https://github.com/huggingface/transformers.git`

Looks like there is an issue finetuning Gemma3 with `eager` attention implementation when installing `transformers` from the main branch

```
{'loss': 0.0, 'grad_norm': nan, 'learning_rate': 7.500000000000001e-06, 'mean_token_accuracy': 0.005072383000515402, 'epoch': 0.03}
```

It works with `SDPA` and `flash_attention_2`, though `eager` is what Gemma3 recommends.

Might be https://github.com/huggingface/transformers/pull/36832, if you are using fp16

> Hi folks 👋
> 
> I'm aware of the problem, but I haven't worked on a solution. Meanwhile, if you set `use_cache: false` in the model config, you should be able to train the model :)

@gante when I try to do this I get this error
```python
TypeError: Gemma3ForConditionalGeneration.__init__() got an unexpected keyword argument 'use_cache'
```