During the first evaluation after training, an OOM (Out of Memory) error occurs.
Author: MrLeiPan
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.43.3
- Platform: Linux-5.15.0-46-generic-x86_64-with-glibc2.31
- Python version: 3.9.19
- Huggingface_hub version: 0.24.2
- Safetensors version: 0.4.3
- Accelerate version: 0.33.0
- PyTorch version (GPU?): 2.4.0 (True)
- GPU type: NVIDIA TITAN V


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction


    train_args = TrainingArguments(
     
        learning_rate=args.lr,
        weight_decay=args.weight_decay,
        per_device_train_batch_size=args.batch_size #4,
        per_device_eval_batch_size=args.batch_size #4,
        log_level=args.log_level,
        logging_steps = 1 ,
        evaluation_strategy=args.evaluation_strategy, # steps
        eval_steps=args.eval_steps,# 2
        eval_accumulation_steps=args.eval_accumulation_steps,#1
    )

    lora_config = LoraConfig(
        r=args.r,
        use_dora=True,
        bias="none",
        lora_alpha=args.lora_alpha,
        target_modules=(
            args.lora_target_modules.split(",")
            if args.lora_target_modules
            else ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
        )
    )
    peft_model = get_peft_model(ehr_model, lora_config)
    trainer = Trainer(
        model=peft_model,
        args=train_args,
        train_dataset=train_data,
        eval_dataset=dev_data,
        tokenizer=tokenizer,
        data_collator=data_collator,
        compute_metrics=compute_metrics,
        callbacks=[info_callback]
    )


### Expected behavior

If no evaluation is performed during training, no OOM (Out of Memory) error occurs. However, I want to conduct evaluations after training for a few epochs. But every time after an evaluation is completed and the next training epoch begins, an OOM error occurs. It seems like the GPU memory usage suddenly increases after backpropagation. What could be the reason?

Comments:
cc @SunMarc @muellerzr 

Im new to using this library.
Im playing around with object detection, and using the example training script. 
https://github.com/huggingface/transformers/blob/e7f4ace0929600606424efd4cd91947bd567d323/examples/pytorch/object-detection/run_object_detection.py

Training finishes fine, gpu memory while training is stable.
But when the final evaluation runs (against the whole test dataset) (with `--do_eval true`) then it seems to just endlessly allocate memory the gpu until it crashes.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@MrLeiPan have you find a solution?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.