An OOM where there should not be any OOM.

### 🐛 Describe the bug

I see similar type of errors being asked about in quite a few places, with advice given being usually useless. The suggestion below to muck around with an environment variable is similarly useless.

What is confounding to me is that memory allocation is tiny in comparison to still available space. Why is this happening?

 
Traceback (most recent call last):
  File "fine-tune.py", line 338, in <module>
    train()
  File "fine-tune.py", line 331, in train
    trainer.train()
  File "/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/transformers/trainer.py", line 1591, in train
    return inner_training_loop(
  File "/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/transformers/trainer.py", line 1726, in _inner_training_loop
    model, self.optimizer = self.accelerator.prepare(self.model, self.optimizer)
  File "/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/accelerate/accelerator.py", line 1280, in prepare
    result = self._prepare_deepspeed(*args)
  File "/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/accelerate/accelerator.py", line 1662, in _prepare_deepspeed
    engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
  File "/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/deepspeed/__init__.py", line 171, in initialize
    engine = DeepSpeedEngine(args=args,
  File "/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 304, in __init__
    self._configure_optimizer(optimizer, model_parameters)
  File "/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1212, in _configure_optimizer
    self.optimizer = self._configure_zero_optimizer(basic_optimizer)
  File "/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/deepspeed/runtime/engine.py", line 1473, in _configure_zero_optimizer
    optimizer = DeepSpeedZeroOptimizer(
  File "/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 509, in __init__
    self.initialize_optimizer_states()
  File "/home/developer/mambaforge/envs/FinGPT/lib/python3.8/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 644, in initialize_optimizer_states
    self.optimizer.step()
  File "/home/developer/pytorch/torch/optim/lr_scheduler.py", line 69, in wrapper
    return wrapped(*args, **kwargs)
  File "/home/developer/pytorch/torch/optim/optimizer.py", line 280, in wrapper
    out = func(*args, **kwargs)
  File "/home/developer/pytorch/torch/optim/optimizer.py", line 33, in _use_grad
    ret = func(self, *args, **kwargs)
  File "/home/developer/pytorch/torch/optim/adamw.py", line 171, in step
    adamw(
  File "/home/developer/pytorch/torch/optim/adamw.py", line 321, in adamw
    func(
  File "/home/developer/pytorch/torch/optim/adamw.py", line 564, in _multi_tensor_adamw
    exp_avg_sq_sqrt = torch._foreach_sqrt(device_exp_avg_sqs)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 134.00 MiB (GPU 0; 11.93 GiB total capacity; 4.48 GiB already allocated; 6.69 GiB free; 4.77 GiB allowed; 4.74 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF


### Versions

UBuntu 20.04
Python 3.8
Anaconda environment
Inside a docker