DPO LoRA loss incorrect with Gradient Accumulation in 4.48.1
Author: winglian
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.48.1
- Platform: Linux-6.8.0-49-generic-x86_64-with-glibc2.35
- Python version: 3.11.10
- Huggingface_hub version: 0.26.3
- Safetensors version: 0.4.5
- Accelerate version: 1.2.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA A40


### Who can help?


See loss charts https://wandb.ai/axolotl-ai/transformers-4_48_1

Likely caused by https://github.com/huggingface/transformers/pull/35651

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

checking out the latest TRL and running:
```bash
python trl/scripts/dpo.py     --dataset_name trl-lib/ultrafeedback_binarized     --model_name_or_path Qwen/Qwen2-0.5B-Instruct     --learning_rate 5.0e-6     --num_train_epochs 1     --per_device_train_batch_size 2     --gradient_accumulation_steps 8     --gradient_checkpointing     --logging_steps 1     --eval_strategy steps     --eval_steps 50     --output_dir Qwen2-0.5B-DPO     --no_remove_unused_columns     --use_peft     --lora_r 32     --lora_alpha 16 --torch_dtype bfloat16 --bf16  --attn_implementation flash_attention_2 --report_to wandb
```
installing older versions of transformers up to 4.48.0 fixes the loss values.
using gradient accumulation of 1 on 4.48.1 seems fine.

### Expected behavior

the loss values using ga=8 should have similar loss values of ga=1

<img width="460" alt="Image" src="https://github.com/user-attachments/assets/92b3c2cd-2439-48a4-95e6-fdaeb9ba4fd1" />

Comments:
hi @winglian, can you check if this is fixed on `main`? We'll be releasing a patch in a couple of days to fix that issue

also broken on main still. Looks like it's maybe a trl issue.

I am using `transformers` directly without `trl` and I'm still seeing a similar issue, so it's not specific to `trl`.
The reported `train/loss` is not divided correctly by `gradient_accumulation_steps`.

I know the loss was fine on `4.44`, but I have tried the following versions and they are all identically incorrect: `4.47.1`, `4.48.0`, `4.48.1`.
The problematic PR (https://github.com/huggingface/transformers/pull/35651) mentioned at the top is only present in `4.48.1`, so it can't have been the cause of the issue.

In the following experiment, the 3 runs are approximately identical except that they use `gradient_accumulation_steps` of 1, 4, and 6. If I divide the train_loss by gradient_accumulation_steps, the issue is solved.
![Image](https://github.com/user-attachments/assets/a340bafb-33d5-4b1a-a51d-f3acadcafec4)

For reference, I am using FSDP. Let me know if you need me to run any more experiments.

@farzadab see the merged trl fix here https://github.com/huggingface/trl/pull/2615/files

Thanks Wing, I can confirm that does solve the issue on my side too!

I still feel like the cause of the issue is just sloppy design on the `transformers` side, not something that should be patched-up downstream. Is it not possible for someone in TRL wanting to handle loss_kwargs (whatever that is) themselves?

Specifically referring to this line:
![Image](https://github.com/user-attachments/assets/b26dfac3-d765-49c1-b014-a24b23c02e87)

It would've been less bug-prone to set `PreTrainedModel.accepts_loss_kwargs = False` and then set it to True only on the models that do use it.