Maybe memory leak occurs after evaluation when using `use_liger_kernel`.
Author: upskyy
Pull request: no
Labels: name: bug

### System Info

`transformers==4.46.1`
`python==3.10.14`

### Who can help?

@muellerzr @SunMarc @ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Thank you for sharing such an amazing project. 

When I used `use_liger_kernel=True` for training, the training memory usage of the Gemma2 model dropped from around 60 GiB to 7 GiB.

However, after running evaluation, the memory usage jumps to 60 GiB. I understand that this is done by accumulating logit values ​​and calculating at the end. **But when resuming training, it doesn't return to the previous memory level, staying at 60 GiB instead.** It seems like there might be a memory leak somewhere.

https://huggingface.co/docs/transformers/main_classes/trainer#transformers.Seq2SeqTrainingArguments.use_liger_kernel


### Expected behavior

The same phenomenon occurred when training with [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory).


Comments:
https://github.com/hiyouga/LLaMA-Factory/issues/6085

Hey @upskyy, thanks for the report ! Could you share a reproducer so that we can investigate faster ? 
cc @ByronHsu did you have that issue in your tests ? 

@SunMarc 
I experimented with LLaMA-Factory, but I think you can reproduce it right away by just `using use_liger_kernel=True` in the huggingface trainer.

Hi @SunMarc @upskyy I believe this might be related to how pytorch mem allocator works. 

1. The allocator allocates the mem for 60GB at the first pass
2. It keeps the mem around for future use unless you explicitly call `empty_cache()`
3. You can print the actually used memory by [cuda.memory_allocated](https://pytorch.org/docs/stable/generated/torch.cuda.memory_allocated.html#torch.cuda.memory_allocated) and the reserved memory by [cuda_memory_reserved](https://pytorch.org/docs/stable/generated/torch.cuda.memory_reserved.html#torch.cuda.memory_reserved). What shown on `nvidia-smi` is the reserved memory.
4. You can also use https://pytorch.org/blog/understanding-gpu-memory-1/#capturing-memory-snapshots to profile.

Also, what matters is the "peak memory", so even if you release by `empty_cache()`, the peak memory will still be 60GB, but it might degrade the performance because `empty_cache()` is expensive. 

Please let me know how it goes. Thanks!

Reference: https://www.perplexity.ai/search/explain-pytorch-gpu-mem-alloca-5PlFr7cjTm2kO3EieFuJbQ#0



@ByronHsu @SunMarc 

Thank you for the detailed explanation!

I understand that the reason it appears in `nvidia-smi` is because both active memory and cached memory are displayed together. The actual memory used for training is less than 60GiB. 
If I do `torch.cuda.empty_cache()`, I can remove the cached memory, but since it is an expensive operation, I understand that I would just cache it and then train, right?

However, in terms of performance, for those who want to train with less memory, shouldn't the cache memory be removed by applying `torch.cuda.empty_cache()`? Or, I think it would be a good idea to leave a log  : )

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.