Is it fine if we do not pass the optimizer through accelerator.prepare() in DDP?
Author: tuvuumass
Pull request: no

Hello!
I remember that in previous versions of Transformers,  the optimizer was not sent to any accelerator device for training (e.g., in older versions of the `run_glue.py` script). So, I wonder if it is fine if we do not pass the optimizer through accelerator.prepare() in DDP in the latest version of Transformers (which means we only send the model and dataloaders to prepare())? This would help solve an OOM issue in my code. 


Comments:
cc @sgugger 

It seems weird that not passing it would avoid an OOM error, as it doesn't do anything special apart handling mixed precision (most of the work is for PyTorch XLA, so mostly for TPUs). For regular GPU training, it should be fine not to send the optimizer, except if you use mixed precision.

Out of curiosity, what is the command giving you an OOM with this script?

@sgugger: You are right, I still got an OOM error after training for a while.

I am fine-tuning a `bert-base` model repeatedly on different datasets (like in continual learning or self-training) using 4 NVIDIA GeForce GTX 1080 Ti GPUs. The procedure is as follows:

```
for i in range(max_iterations):
    finetuning_procedure
```

The `finetuning_procedure` is similar to `run_glue_no_trainer.py` that uses accelerator and DDP.  If I call it from a bash script, everything works perfectly. However, when I convert it into a python function and call it within the for loop, I keep getting OOM issues after training for several iterations.

```
RuntimeError: CUDA out of memory. Tried to allocate X MiB (GPU A; B GiB total capacity; C GiB already allocated; D MiB free; E MiB cached)
```

I figured out the problem is that the memory accumulates across iterations and at some point, I’ll get an OOM error. Calling from a bash script works well because it clears up the memory after the script gets done.

I found this thread https://github.com/huggingface/transformers/issues/1742, which suggests deleting all tensor objects after each iteration then collecting garbage and emptying the GPU cache:

```
delete objects
gc.collect()
torch.cuda.empty_cache()
```
 
It helped a bit but I still got an OOM error after training for like 10 iterations. Then, I tried another trick that moves the tensor objects to CPU before deleting them based on this thread https://github.com/pytorch/pytorch/issues/31252

```
gc.collect()
for obj in gc.get_objects():
  if not isinstance(obj, torch.Tensor):
    continue

  obj.data = obj.data.cpu()
  if isinstance(obj, torch.nn.Parameter) and obj.grad is not None:
    obj.grad.data = obj.grad.cpu()

  del obj

torch.cuda.empty_cache()
```
It helped a lot and I was able to train up to 40 iterations, but then I got the following error

```
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 1 (pid: 137092) of binary: /home/tuvu/anaconda3/envs/bert/bin/python
Traceback (most recent call last):
  File "/home/tuvu/anaconda3/envs/bert/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/tuvu/anaconda3/envs/bert/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/tuvu/anaconda3/envs/bert/lib/python3.8/site-packages/torch/distributed/launch.py", line 193, in <module>
    main()
  File "/home/tuvu/anaconda3/envs/bert/lib/python3.8/site-packages/torch/distributed/launch.py", line 189, in main
    launch(args)
  File "/home/tuvu/anaconda3/envs/bert/lib/python3.8/site-packages/torch/distributed/launch.py", line 174, in launch
    run(args)
  File "/home/tuvu/anaconda3/envs/bert/lib/python3.8/site-packages/torch/distributed/run.py", line 710, in run
    elastic_launch(
  File "/home/tuvu/anaconda3/envs/bert/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 131, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/tuvu/anaconda3/envs/bert/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 259, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
=======================================================
selftraining.py FAILED
-------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
-------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2022-02-17_11:54:34
  host      : node132.cm.cluster
  rank      : 1 (local_rank: 1)
  exitcode  : -9 (pid: 137092)
  error_file: <N/A>
  traceback : Signal 9 (SIGKILL) received by PID 137092
```

It did not throw up an OOM error like before but I feel like it's still about memory (exitcode: -9).

@sgugger Any insight on this would be much appreciated! Thanks! 

I would strongly recommend not doing a loop on a function for training but launching various scripts (which can be automated in a bash loop as well), as it's very hard to keep track of all objects that Python might allocate memory for and not release.

The `Accelerator` has a method to clean up a bit its internal references, but the code you suggested should already clean up everything, so I don't have a better suggestion.

@sgugger: Gotcha! Thanks for the suggestion!

> 

I'm running into a similar issue with memory growing across iterations. Obviously, your suggestion speaks directly to this. Can I ask why you think the several-scripts method is better? I mean, if it works, that's an argument by itself. But I'd be curious to hear if you have further thoughts.

FYI, I fixed my memory issue by using the [accelerator.free_memory()](https://huggingface.co/docs/accelerate/accelerator#accelerate.Accelerator.free_memory) call at the end of my method. Hopefully, this will help future people searching for memory issues when using accelerators

Yes, [accelerator.free_memory()](https://huggingface.co/docs/accelerate/accelerator#accelerate.Accelerator.free_memory) also solved the problem for me.

@sgugger: Hi Sylvain, this pull request https://github.com/huggingface/transformers/pull/16738 implements self-training for text-classification tasks. It uses [accelerator.free_memory()](https://huggingface.co/docs/accelerate/accelerator#accelerate.Accelerator.free_memory) to release all references to the internal objects stored and call the garbage collector. Could you help review when you have a chance?