Resumption of training using the same parameters as before fails with CUDA out of memory error.
Author: YISTANFORD
Pull request: no
Labels: name: bug

### System Info

```shell
- `transformers` version: 4.15.0
- Platform: Linux-3.10.0-1160.59.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core
- Python version: 3.6.8
- PyTorch version (GPU?): 1.7.0 (False)
- Tensorflow version (GPU?): 2.6.2 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
```


### Who can help?

@ydshieh
@sgugger

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. I used longformer_training.py to train longformer, without getting a CUDA out of memory error, the training continues until the server time limit is reached.
[longformer_training.txt](https://github.com/huggingface/transformers/files/8646199/longformer_training.txt)

2. I rented a server again and resumed longformer training using longformer_resuming_training.py. For some reason I got a CUDA out of memory error.
[longformer_resume_training.txt](https://github.com/huggingface/transformers/files/8646200/longformer_resume_training.txt)

Errors are as follows.
[error.txt](https://github.com/huggingface/transformers/files/8646204/error.txt)



### Expected behavior

```shell
I am using Huggingface to research patent documents and I am really in need of help as my term is coming to an end. Your help would be really appreciated.
The expected behavior is that the same model is trained before and after the training is resumed, so it will not CUDA out of memory after resumption and complete the training successfully.
Thank you very much in advance.
```


Comments:
Hi, @YISTANFORD 

My colleague sgugger might have better ideas. But here are some suggestions from my side:

- It's probably a good idea to work with newer versions.
  - Python 3.6 is EOL (in Dec 2021)
  - PyTorch is now 1.11, but 1.10 might be good too
  - TensorFlow, `transformers`, etc.
  - (Of course, I understand you might have some constraints in your work.)
- It would be very helpful if you could try to 
  - measure the GPU memory allocated before the training starts (i.e. before the line `for epoch in range(...)` in `train()`)
    - check this GPU memory usage in the original train and the one resumed from a checkpoint to see if there is any difference. 
  - You might also want to try something like `torch.cuda.empty_cache()` (at some places, but I am not sure where at this moment).

I also found that the `REAL_BS` and `ACCUM_NUM` are different in the 2 scripts. But I guess you tried to reduce the GPU memory usage by using gradient accumulation.
 

Hello @ydshieh ,

Thank you for your reply.
I really appreciate all the tips and tricks.
May I try another version and measure GPU memory usage and report back to you?
I would like to version up python to 3.7 first and pytorch to 1.10.
By the way, I apologize for my lack of knowledge, but I was wondering if you know of any way to measure GPU memory usage at each step of the code.
Thank you very much again.


Hi @YISTANFORD,  

You can check  https://pytorch.org/docs/stable/cuda.html#torch.cuda.memory_stats, I would probably try both
- [memory_reserved](https://pytorch.org/docs/stable/generated/torch.cuda.memory_reserved.html#torch.cuda.memory_reserved)
- [memory_allocated](https://pytorch.org/docs/stable/generated/torch.cuda.memory_allocated.html#torch.cuda.memory_allocated)

You can also use `nvidia-smi` to get a bit more information, by calling bash command in Python using `os` module.

BTW, if possible, maybe with Python 3.8 or 3.9. But go ahead with what you could do :-)

Thank you!

Hello @ydshieh ,

Thank you very much for your kind guidance.
I will try python with 3.8 or 3.9.
Can I report back when I get the results?
Thank you very much again.

> Can I report back when I get the results? 

Sure!

> Hello @ydshieh ,
> 
> Thank you for your reply. I really appreciate all the tips and tricks. May I try another version and measure GPU memory usage and report back to you? I would like to version up python to 3.7 first and pytorch to 1.10. By the way, I apologize for my lack of knowledge, but I was wondering if you know of any way to measure GPU memory usage at each step of the code. Thank you very much again.

You can also use something like [WandB](https://wandb.ai/site/articles/monitor-improve-gpu-usage-for-model-training) to track your gpu utilization over time. 

Also why do the scripts use different batch sizes and grad accumulations? 


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.