When I use BF16 or FP16 to perform Lora fine-tuning on GemMA-3-12B-it, there will be an error when saving the checkpoint, but FP32 is normal
Author: Fluchw
Pull request: no
Labels: name: bug

### System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.50.0.dev0
- Platform: Linux-5.15.0-43-generic-x86_64-with-glibc2.35
- Python version: 3.11.7
- Huggingface_hub version: 0.29.3
- Safetensors version: 0.5.3
- Accelerate version: 1.5.2
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 4090



### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```text
llamafactory-cli train \
    --stage sft \
    --do_train True \
    --model_name_or_path /data/llm/models/gemma-3-12b-it \
    --preprocessing_num_workers 16 \
    --finetuning_type lora \
    --template gemma3 \
    --flash_attn auto \
    --dataset_dir ../dataset/ocCuts \
    --dataset group_0 \
    --cutoff_len 2048 \
    --learning_rate 1e-05 \
    --num_train_epochs 1.0 \
    --max_samples 100 \
    --per_device_train_batch_size 1 \
    --gradient_accumulation_steps 8 \
    --lr_scheduler_type cosine \
    --max_grad_norm 1.0 \
    --logging_steps 1 \
    --save_steps 50 \
    --warmup_steps 0 \
    --packing False \
    --report_to none \
    --use_swanlab True \
    --output_dir saves/Gemma-3-12B-Instruct/lora/occ1 \
    --bf16 True \
    --plot_loss True \
    --trust_remote_code True \
    --ddp_timeout 180000000 \
    --include_num_input_tokens_seen True \
    --optim adamw_torch \
    --quantization_bit 8 \
    --quantization_method bitsandbytes \
    --double_quantization True \
    --lora_rank 8 \
    --lora_alpha 16 \
    --lora_dropout 0 \
    --lora_target all \
    --swanlab_project llamafactory \
    --swanlab_api_key LvdI91Ym9nBEQ0QvjKTRp \
    --swanlab_mode cloud \
    --val_size 0.2 \
    --eval_strategy steps \
    --eval_steps 50 \
    --per_device_eval_batch_size 1
```

```text
 File "/data/llm/LLaMA-Factory/.venv/bin/llamafactory-cli", line 10, in <module>                   
    sys.exit(main())
             ^^^^^^
  File "/data/llm/LLaMA-Factory/src/llamafactory/cli.py", line 118, in main
    run_exp()
  File "/data/llm/LLaMA-Factory/src/llamafactory/train/tuner.py", line 103, in run_exp
    _training_function(config={"args": args, "callbacks": callbacks})
  File "/data/llm/LLaMA-Factory/src/llamafactory/train/tuner.py", line 68, in _training_function
    run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
  File "/data/llm/LLaMA-Factory/src/llamafactory/train/sft/workflow.py", line 120, in run_sft
    metrics = trainer.evaluate(metric_key_prefix="eval", **gen_kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/llm/LLaMA-Factory/.venv/lib/python3.11/site-packages/transformers/trainer_seq2seq.py", line 197, in evaluate
    return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/llm/LLaMA-Factory/.venv/lib/python3.11/site-packages/transformers/trainer.py", line 4118, in evaluate
    output = eval_loop(
             ^^^^^^^^^^
  File "/data/llm/LLaMA-Factory/.venv/lib/python3.11/site-packages/transformers/trainer.py", line 4312, in evaluation_loop
    losses, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/llm/LLaMA-Factory/src/llamafactory/train/sft/trainer.py", line 114, in prediction_step
    loss, generated_tokens, _ = super().prediction_step(
                                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/llm/LLaMA-Factory/.venv/lib/python3.11/site-packages/transformers/trainer_seq2seq.py", line 295, in prediction_step
    return super().prediction_step(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/llm/LLaMA-Factory/.venv/lib/python3.11/site-packages/transformers/trainer.py", line 4528, in prediction_step
    loss, outputs = self.compute_loss(model, inputs, return_outputs=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/llm/LLaMA-Factory/.venv/lib/python3.11/site-packages/transformers/trainer.py", line 3772, in compute_loss
    outputs = model(**inputs)
              ^^^^^^^^^^^^^^^
  File "/data/llm/LLaMA-Factory/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/llm/LLaMA-Factory/.venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/llm/LLaMA-Factory/.venv/lib/python3.11/site-packages/accelerate/utils/operations.py", line 819, in forward
    return model_forward(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/llm/LLaMA-Factory/.venv/lib/python3.11/site-packages/accelerate/utils/operations.py", line 807, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/llm/LLaMA-Factory/.venv/lib/python3.11/site-packages/accelerate/utils/operations.py", line 786, in convert_to_fp32
    return recursively_apply(_convert_to_fp32, tensor, test_type=_is_fp16_bf16_tensor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/llm/LLaMA-Factory/.venv/lib/python3.11/site-packages/accelerate/utils/operations.py", line 118, in recursively_apply
    {
  File "/data/llm/LLaMA-Factory/.venv/lib/python3.11/site-packages/accelerate/utils/operations.py", line 119, in <dictcomp>
    k: recursively_apply(
       ^^^^^^^^^^^^^^^^^^
  File "/data/llm/LLaMA-Factory/.venv/lib/python3.11/site-packages/accelerate/utils/operations.py", line 126, in recursively_apply
    return func(data, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/llm/LLaMA-Factory/.venv/lib/python3.11/site-packages/accelerate/utils/operations.py", line 778, in _convert_to_fp32
    return tensor.float()
           ^^^^^^^^^^^^
'HybridCache' object has no attribute 'float'
```

### Expected behavior

I don't know if this is my problem or his problem

Comments:
Oh, hybrid cache 🙃 

cc @gante for cache if you have time to look

Hey @Fluchw 👋 

The shared script uses tools outside `transformers`, so I can only try to guess what's truly happening. At a first glance, generation cache is being used at training time, which should happen (especially with gradient accumulation 👀 )

Can you try to set `use_cache: False` in `/data/llm/models/gemma-3-12b-it`? If that doesn't work, I'm afraid I'll need a reproduction script using `Trainer` (as opposed to `llamafactory-cli`)

> 嘿[@Fluchw](https://github.com/Fluchw)👋
> 
> 共享脚本使用了 之外的工具`transformers`，因此我只能尝试猜测真正发生了什么。乍一看，在训练时使用了生成缓存，这应该会发生（尤其是在梯度累积的情况下 👀 ）
> 
> 您可以尝试设置`use_cache: False`吗`/data/llm/models/gemma-3-12b-it`？如果不起作用，恐怕我需要使用`Trainer`（而不是`llamafactory-cli`）的复制脚本

Solved, modify the model's config. json file

```
"text_config": {
"hidden_size": 3840,
"intermediate_size": 15360,
"model_type": "gemma3_text",
"num_attention_heads": 16,
"num_hidden_layers": 48,
"num_key_value_heads": 8,
"rope_scaling": {
"factor": 8.0,
"rope_type": "linear"
},
"sliding_window": 1024
},
```
Add `'use_cache': false`
 It can solve this problem
```
"text_config": {
"use_cache": false,
"hidden_size": 3840,
"intermediate_size": 15360,
"model_type": "gemma3_text",
"num_attention_heads": 16,
"num_hidden_layers": 48,
"num_key_value_heads": 8,
"rope_scaling": {
"factor": 8.0,
"rope_type": "linear"
},
"sliding_window": 1024
},
```

@Fluchw the PR above should fix this issue 🤗 

> [@Fluchw](https://github.com/Fluchw) the PR above should fix this issue 🤗上面的 PR 应该可以解决这个问题 🤗

Great, thank you🤗