Out of Memory Error (OOM) only during evaluation phase of run_lm_finetuning.py and run_glue.py
Author: CMobley7
Pull request: no
Labels: name: wontfix

## 🐛 Bug

<!-- Important information -->

Model I am using (Bert, XLNet....): roberta-large

Language I am using the model on (English, Chinese....): English

The problem arise when using:
* the official example scripts: run_lm_finetuning.py and run_glue.py

The tasks I am working on is:
* my own task or dataset: my own dataset which I have put into the correct format for `run_lm_finetuning` and the `SST-2` task for `run_glue.py`

## To Reproduce

When I use the official `run_lm_finetuning.py` and `run_glue.py` scripts with `roberta-large` on my own data, I run into out of memory (`OOM`) issues during `evaluation`. `Training completes successfully` with a `batch_size` of `1` and a `max_senquence_length` of `512`. However, when `evaluating` the 4th or 5th checkpoint, I get an `OOM` error. I tested using `distributed` training with `FP16` on `2 Tesla P100-SXM2 16 GB` and `2 Tesla P100-PCIE 12 GB` with the same results. I'm currently testing on `2 Tesla M40 24GB`; however, I don't always have access to these GPUs; so,  this is untenable long term.

## Expected behavior

Since it finishes training and evaluation of the first few checkpoints with the aforementioned setting, I would expect it to be able to evaluate every checkpoint without error. However, it seems that there is a bug in the evaluation code. I assume it is not properly clearing the memory after testing each checkpoint. 

## Environment

* OS: Ubuntu 18.04
* Python version: 3.6.8
* PyTorch version: 1.3
* PyTorch Transformers version (or branch): 2.1.1
* Using GPU Yes,  Tesla M40 24GB, Tesla P100-SXM2 16 GB, Tesla P100-PCIE 12 GB
* Distributed of parallel setup 2x of each separate GPU


Comments:
Have you tried the following lines after the call to the `evaluate` function (in the loop over checkpoints)?

```python
del model
torch.cuda.empty_cache()
```

Let me know if this works.

@rlouf, I changed

```
# Evaluation
    results = {}
    if args.do_eval and args.local_rank in [-1, 0]:
        checkpoints = [args.output_dir]
        if args.eval_all_checkpoints:
            checkpoints = list(os.path.dirname(c) for c in sorted(glob.glob(args.output_dir + '/**/' + WEIGHTS_NAME, recursive=True)))
            logging.getLogger("transformers.modeling_utils").setLevel(logging.WARN)  # Reduce logging
        logger.info("Evaluate the following checkpoints: %s", checkpoints)
        for checkpoint in checkpoints:
            global_step = checkpoint.split('-')[-1] if len(checkpoints) > 1 else ""
            prefix = checkpoint.split('/')[-1] if checkpoint.find('checkpoint') != -1 else ""
            
            model = model_class.from_pretrained(checkpoint)
            model.to(args.device)
            result = evaluate(args, model, tokenizer, prefix=prefix)
            result = dict((k + '_{}'.format(global_step), v) for k, v in result.items())
            results.update(result)

    return results
```
to
```
# Evaluation
    results = {}
    if args.do_eval and args.local_rank in [-1, 0]:
        checkpoints = [args.output_dir]
        if args.eval_all_checkpoints:
            checkpoints = list(os.path.dirname(c) for c in sorted(glob.glob(args.output_dir + '/**/' + WEIGHTS_NAME, recursive=True)))
            logging.getLogger("transformers.modeling_utils").setLevel(logging.WARN)  # Reduce logging
        logger.info("Evaluate the following checkpoints: %s", checkpoints)
        for checkpoint in checkpoints:
            global_step = checkpoint.split('-')[-1] if len(checkpoints) > 1 else ""
            prefix = checkpoint.split('/')[-1] if checkpoint.find('checkpoint') != -1 else ""
            
            model = model_class.from_pretrained(checkpoint)
            model.to(args.device)
            result = evaluate(args, model, tokenizer, prefix=prefix)
            del model
            torch.cuda.empty_cache()
            result = dict((k + '_{}'.format(global_step), v) for k, v in result.items())
            results.update(result)

    return results

```
which didn't help.

So, I looked through https://github.com/huggingface/transformers/issues/1742 and added

```
del model
gc.collect()
torch.cuda.empty_cache()
```

This didn't work either; so, I change the scheduler to the following while leaving

```
del model
gc.collect()
torch.cuda.empty_cache()
```

```
def warmup_linear_schedule(optimizer, warmup_steps, t_total, last_epoch=-1):
    def lr_lambda(step):
        if step < warmup_steps:
            return float(step) / float(max(1, warmup_steps))
        return max(0.0, float(t_total - step) / float(max(1.0, t_total - warmup_steps)))

    return LambdaLR(optimizer, lr_lambda, last_epoch=-1)

```
This hasn't worked either.

Any ideas? It doesn't seem like I can `del model`

Hi! Changing the scheduler should not do anything since you are in evaluation mode. This is likely not due to the model either, since `del` + `gc.collect()` should remove it from memory. Could you look at by how much the memory increases after each step?

So, when using `roberta-large` the memory increases by ~2.25 GB after each checkpoint starting at ~7GB.

Can you have a look at which tensors are still being tracked, using [this code](https://github.com/huggingface/transformers/issues/1742#issuecomment-550327172) after your del statement?

My apologies for the delay in my answer. I had hoped that https://github.com/huggingface/transformers/commit/9629e2c676efab31d1f27a3f27d811e91fd05a92 would solve my issue; however, evaluation doesn't appear to be distributed. So, it is still using one card and is still ~2.25 GB after each checkpoint. Should I open up a separate issue for the distributed evaluation issue? After adding 
```
def print_gpu_obj():
    import gc
    GPU_count = 0
    Pinned_count = 0
    for tracked_object in gc.get_objects():
        if torch.is_tensor(tracked_object):
            if tracked_object.is_cuda:
                GPU_count+=1
            if tracked_object.is_pinned():
                Pinned_count+=1
            

    print("There are {} cuda objects".format(GPU_count))
    print("There are {} pinned objects".format(Pinned_count))
```
to the code, I get

before `del model`
checkpoint 0
```
There are 1087 cuda objects
There are 0 pinned objects
```
checkpoint 1
```
There are 1673 cuda objects
There are 0 pinned objects
```
checkpoint 2
```
There are 2259 cuda objects
There are 0 pinned objects
```
checkpoint 3
```
There are 2845 cuda objects
There are 0 pinned objects
```

after `del model` and before `gc.collect`
checkpoint 0
```
There are 984 cuda objects
There are 0 pinned objects
```
checkpoint 1
```
There are 1570 cuda objects
There are 0 pinned objects
```
checkpoint 2
```
There are 2156 cuda objects
There are 0 pinned objects
```
checkpoint 3
```
There are 2742 cuda objects
There are 0 pinned objects
```

after `gc.collect` and before `torch.cuda.empty_cache()`  and after `torch.cuda.empty_cache()` are both the same as after `del model` and before `gc.collect`.

It appears that 586 objects are added at each checkpoint, which cannot be deleted and unallocated. Any suggestions?

This looks like some data is kept in memory after evaluation. Any way you could try to track this down?

My apologies for the delay in getting back to this, as well as my ignorance on this subject, but how would I track this down. As in what commands or tools should I use? Again sorry for the late reply, I got stuck on another project for a bit.

I have a similar issue. During training with run_lm_finetuning.py, the memory usage of a single GPU  is 30G/32G. And after the training stage, I mean at the beginning of eval, the memory doesn't drop down and the evaluation stage is always getting OOM. If I add torch.cuda.empty_cache() before evaluation, the memory drops down to 22G, which means there are still 22GB tensors exist but I don't know where they are and how to free them. If I do the evaluation with run_lm_finetuning.py without training, everything is ok and the memory usage is about 8G.

> I have a similar issue. During training with run_lm_finetuning.py, the memory usage of a single GPU is 30G/32G. And after the training stage, I mean at the beginning of eval, the memory doesn't drop down and the evaluation stage is always getting OOM. If I add torch.cuda.empty_cache() before evaluation, the memory drops down to 22G, which means there are still 22GB tensors exist but I don't know where they are and how to free them. If I do the evaluation with run_lm_finetuning.py without training, everything is ok and the memory usage is about 8G.

PS: If I set --evaluate_during_training, there is no OOM issue.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I ran into a very similar issue while working with transformers==4.50.3 on a ~500k example evaluation run using bert-base-cased. The crash occurred consistently at around 94–95%, with GPU memory pegging at ~3.75 GB regardless of batch size or cache clearing.

I ended up writing up the full environment, reproduction steps, and mitigation that worked for me in a separate [issue](https://github.com/knotreeltest/no-repo/issues/1).

Hope this helps anyone running into the same failure!